2024-04-24T15:45:12.461 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T15:45:12.462 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/compare-mirror-image-alternate-primary-nbd} --name yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592 -- /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592/orig.config.yaml 2024-04-24T15:45:12.503 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T15:45:12.590 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592 branch: wip-yuriw4-testing-20240423.151325-quincy description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/compare-mirror-image-alternate-primary-nbd} email: yweinste@redhat.com first_in_suite: false job_id: '7671592' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each name: yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuriw4-testing-20240423.151325-quincy ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: extra_packages: - rbd-nbd extra_system_packages: - pv flavor: default sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuriw4-testing-20240423.151325-quincy sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9954 sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuriw4-testing-20240423.151325-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 targets: smithi042.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOgMBWGhQgRrZqrK+UViDZS8uTGr6VLDef+vszWRP2HNdvqzjekW/QRf9O1n2zFUkO6rlFsgHaI+6BgdEuX4NRs= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBohcSRsl/owVXyMLt+NRae4wF031BSohDnw1SyHhey5bNWqKav/KxJtJdv2MOtxEdgw8B9hOglR8UDq4eq9vg= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/compare_mirror_image_alternate_primary.sh env: RBD_DEVICE_TYPE: nbd timeout: 3h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_14:09:24 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T15:45:12.590 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa; will attempt to use it 2024-04-24T15:45:12.591 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa/tasks 2024-04-24T15:45:12.591 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T15:45:12.592 INFO:teuthology.task.internal:Checking packages... 2024-04-24T15:45:12.614 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '330067dbce47655d1f762ef14d546d2fde16a6a9' 2024-04-24T15:45:12.614 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T15:45:12.615 INFO:teuthology.packaging:ref: None 2024-04-24T15:45:12.615 INFO:teuthology.packaging:tag: None 2024-04-24T15:45:12.615 INFO:teuthology.packaging:branch: wip-yuriw4-testing-20240423.151325-quincy 2024-04-24T15:45:12.615 INFO:teuthology.packaging:sha1: 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:45:12.615 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuriw4-testing-20240423.151325-quincy 2024-04-24T15:45:12.831 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-915.g330067db 2024-04-24T15:45:12.832 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T15:45:12.839 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T15:45:12.839 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T15:45:12.882 INFO:teuthology.task.internal:Saving configuration 2024-04-24T15:45:12.898 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T15:45:12.904 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T15:45:12.926 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 15:40:13.371745', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOgMBWGhQgRrZqrK+UViDZS8uTGr6VLDef+vszWRP2HNdvqzjekW/QRf9O1n2zFUkO6rlFsgHaI+6BgdEuX4NRs='} 2024-04-24T15:45:12.946 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 15:40:13.369935', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBohcSRsl/owVXyMLt+NRae4wF031BSohDnw1SyHhey5bNWqKav/KxJtJdv2MOtxEdgw8B9hOglR8UDq4eq9vg='} 2024-04-24T15:45:12.947 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T15:45:12.954 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-04-24T15:45:12.954 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-04-24T15:45:12.954 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T15:45:13.017 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa2f5c27160>, signals=[15]) 2024-04-24T15:45:13.017 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T15:45:13.023 INFO:teuthology.task.internal:Opening connections... 2024-04-24T15:45:13.023 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi042.front.sepia.ceph.com 2024-04-24T15:45:13.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T15:45:13.101 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T15:45:13.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T15:45:13.175 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T15:45:13.183 DEBUG:teuthology.orchestra.run.smithi042:> uname -m 2024-04-24T15:45:13.204 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2024-04-24T15:45:13.205 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2024-04-24T15:45:13.266 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="Red Hat Enterprise Linux" 2024-04-24T15:45:13.266 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="8.6 (Ootpa)" 2024-04-24T15:45:13.266 INFO:teuthology.orchestra.run.smithi042.stdout:ID="rhel" 2024-04-24T15:45:13.266 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE="fedora" 2024-04-24T15:45:13.266 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="8.6" 2024-04-24T15:45:13.266 INFO:teuthology.orchestra.run.smithi042.stdout:PLATFORM_ID="platform:el8" 2024-04-24T15:45:13.266 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-24T15:45:13.266 INFO:teuthology.orchestra.run.smithi042.stdout:ANSI_COLOR="0;31" 2024-04-24T15:45:13.266 INFO:teuthology.orchestra.run.smithi042.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-24T15:45:13.267 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://www.redhat.com/" 2024-04-24T15:45:13.267 INFO:teuthology.orchestra.run.smithi042.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-24T15:45:13.267 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T15:45:13.267 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:45:13.267 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-24T15:45:13.267 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-24T15:45:13.267 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-24T15:45:13.267 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-24T15:45:13.268 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2024-04-24T15:45:13.287 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-04-24T15:45:13.308 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-04-24T15:45:13.308 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Red Hat Enterprise Linux" 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8.6 (Ootpa)" 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:ID="rhel" 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="fedora" 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8.6" 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8" 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.redhat.com/" 2024-04-24T15:45:13.368 INFO:teuthology.orchestra.run.smithi136.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-04-24T15:45:13.369 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T15:45:13.369 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:45:13.369 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-24T15:45:13.369 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-04-24T15:45:13.369 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-04-24T15:45:13.369 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-04-24T15:45:13.369 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-04-24T15:45:13.387 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T15:45:13.397 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T15:45:13.405 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T15:45:13.405 DEBUG:teuthology.orchestra.run.smithi042:> test '!' -e /home/ubuntu/cephtest 2024-04-24T15:45:13.407 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-04-24T15:45:13.426 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T15:45:13.433 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T15:45:13.433 DEBUG:teuthology.orchestra.run.smithi042:> test -z $(ls -A /var/lib/ceph) 2024-04-24T15:45:13.469 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-04-24T15:45:13.495 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T15:45:13.554 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T15:45:13.576 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-24T15:45:13.577 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-24T15:45:13.577 DEBUG:teuthology.orchestra.run.smithi042:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T15:45:13.578 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T15:45:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:45:13.597 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-04-24T15:45:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:45:13.599 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-04-24T15:45:13.658 INFO:teuthology.orchestra.run.smithi136.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:45:13.658 INFO:teuthology.task.kernel:Running kernel on smithi136: 4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:45:13.658 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-04-24T15:45:13.660 INFO:teuthology.orchestra.run.smithi042.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:45:13.661 INFO:teuthology.task.kernel:Running kernel on smithi042: 4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:45:13.661 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum install -y kernel 2024-04-24T15:45:15.152 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-04-24T15:45:15.152 INFO:teuthology.orchestra.run.smithi136.stdout:Unable to read consumer identity 2024-04-24T15:45:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:45:15.393 INFO:teuthology.orchestra.run.smithi136.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:45:15.393 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:45:15.540 INFO:teuthology.orchestra.run.smithi042.stdout:Updating Subscription Management repositories. 2024-04-24T15:45:15.540 INFO:teuthology.orchestra.run.smithi042.stdout:Unable to read consumer identity 2024-04-24T15:45:15.837 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:45:15.837 INFO:teuthology.orchestra.run.smithi042.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:45:15.837 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:45:17.067 INFO:teuthology.orchestra.run.smithi136.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-04-24T15:45:17.717 INFO:teuthology.orchestra.run.smithi042.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-04-24T15:45:19.675 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 16 MB 00:02 2024-04-24T15:45:20.734 INFO:teuthology.orchestra.run.smithi042.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 16 MB 00:02 2024-04-24T15:45:23.292 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 455 kB/s | 24 kB 00:00 2024-04-24T15:45:24.044 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-24T15:45:24.044 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-24T15:45:24.109 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-04-24T15:45:24.110 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-04-24T15:45:24.110 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-04-24T15:45:24.236 DEBUG:teuthology.orchestra.run.smithi136:> echo no | sudo yum reinstall kernel || true 2024-04-24T15:45:24.573 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-04-24T15:45:24.573 INFO:teuthology.orchestra.run.smithi136.stdout:Unable to read consumer identity 2024-04-24T15:45:24.718 INFO:teuthology.orchestra.run.smithi042.stdout:lab-extras 447 kB/s | 24 kB 00:00 2024-04-24T15:45:24.810 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:45:24.810 INFO:teuthology.orchestra.run.smithi136.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:45:24.810 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:45:25.121 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:02 ago on Wed 24 Apr 2024 03:45:23 PM UTC. 2024-04-24T15:45:25.214 INFO:teuthology.orchestra.run.smithi136.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-24T15:45:25.216 INFO:teuthology.orchestra.run.smithi136.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-24T15:45:25.251 INFO:teuthology.orchestra.run.smithi136.stderr:Error: No packages marked for reinstall. 2024-04-24T15:45:25.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum reinstall -y kernel || true 2024-04-24T15:45:25.530 INFO:teuthology.orchestra.run.smithi042.stdout:Package kernel-4.18.0-513.9.1.el8_9.x86_64 is already installed. 2024-04-24T15:45:25.530 INFO:teuthology.orchestra.run.smithi042.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2024-04-24T15:45:25.630 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-04-24T15:45:25.632 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-04-24T15:45:25.632 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-04-24T15:45:25.633 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-04-24T15:45:25.633 INFO:teuthology.orchestra.run.smithi136.stdout:Unable to read consumer identity 2024-04-24T15:45:25.783 DEBUG:teuthology.orchestra.run.smithi042:> echo no | sudo yum reinstall kernel || true 2024-04-24T15:45:25.834 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:45:25.834 INFO:teuthology.orchestra.run.smithi136.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:45:25.834 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:45:26.139 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 03:45:23 PM UTC. 2024-04-24T15:45:26.173 INFO:teuthology.orchestra.run.smithi042.stdout:Updating Subscription Management repositories. 2024-04-24T15:45:26.173 INFO:teuthology.orchestra.run.smithi042.stdout:Unable to read consumer identity 2024-04-24T15:45:26.202 INFO:teuthology.orchestra.run.smithi136.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-24T15:45:26.202 INFO:teuthology.orchestra.run.smithi136.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-24T15:45:26.242 INFO:teuthology.orchestra.run.smithi136.stderr:Error: No packages marked for reinstall. 2024-04-24T15:45:26.289 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T15:45:26.378 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:45:26.378 INFO:teuthology.orchestra.run.smithi042.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:45:26.378 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:45:26.446 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:45:26.447 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:45:26.447 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T15:45:26.447 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-04-24T15:45:26.447 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T15:45:26.477 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-04-24T15:45:26.489 DEBUG:teuthology.parallel:result is None 2024-04-24T15:45:26.746 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:00:02 ago on Wed 24 Apr 2024 03:45:24 PM UTC. 2024-04-24T15:45:26.827 INFO:teuthology.orchestra.run.smithi042.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-24T15:45:26.834 INFO:teuthology.orchestra.run.smithi042.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-24T15:45:26.897 INFO:teuthology.orchestra.run.smithi042.stderr:Error: No packages marked for reinstall. 2024-04-24T15:45:26.954 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum reinstall -y kernel || true 2024-04-24T15:45:27.337 INFO:teuthology.orchestra.run.smithi042.stdout:Updating Subscription Management repositories. 2024-04-24T15:45:27.338 INFO:teuthology.orchestra.run.smithi042.stdout:Unable to read consumer identity 2024-04-24T15:45:27.586 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:45:27.586 INFO:teuthology.orchestra.run.smithi042.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-04-24T15:45:27.587 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:45:27.949 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 03:45:24 PM UTC. 2024-04-24T15:45:28.018 INFO:teuthology.orchestra.run.smithi042.stdout:Installed package kernel-4.18.0-513.9.1.el8_9.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2024-04-24T15:45:28.018 INFO:teuthology.orchestra.run.smithi042.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2024-04-24T15:45:28.078 INFO:teuthology.orchestra.run.smithi042.stderr:Error: No packages marked for reinstall. 2024-04-24T15:45:28.135 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T15:45:28.316 INFO:teuthology.orchestra.run.smithi042.stdout:kernel-4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:45:28.317 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.9.1.el8_9.x86_64 2024-04-24T15:45:28.317 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T15:45:28.317 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-04-24T15:45:28.317 DEBUG:teuthology.orchestra.run.smithi042:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T15:45:28.355 INFO:teuthology.orchestra.run.smithi042.stdout:ttyS1 2024-04-24T15:45:28.371 DEBUG:teuthology.parallel:result is None 2024-04-24T15:45:28.371 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T15:45:28.378 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T15:45:28.379 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T15:45:28.413 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T15:45:28.434 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T15:45:28.440 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T15:45:28.447 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T15:45:28.447 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T15:45:28.473 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T15:45:28.508 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T15:45:28.514 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T15:45:28.515 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T15:45:28.540 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T15:45:28.577 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T15:45:28.580 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T15:45:28.590 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T15:45:28.593 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T15:45:28.594 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T15:45:28.601 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T15:45:28.602 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T15:45:28.636 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T15:45:28.669 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T15:45:28.677 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T15:45:28.677 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T15:45:28.712 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T15:45:28.732 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T15:45:28.821 DEBUG:teuthology.orchestra.run.smithi042:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T15:45:28.889 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T15:45:28.937 DEBUG:teuthology.orchestra.run.smithi042:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T15:45:28.988 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:45:28.989 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T15:45:29.063 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T15:45:29.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T15:45:29.166 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T15:45:29.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T15:45:29.300 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:45:29.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T15:45:29.369 DEBUG:teuthology.orchestra.run.smithi042:> sudo service rsyslog restart 2024-04-24T15:45:29.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-04-24T15:45:29.430 INFO:teuthology.orchestra.run.smithi042.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T15:45:29.453 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T15:45:29.857 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T15:45:29.864 INFO:teuthology.task.internal:Starting timer... 2024-04-24T15:45:29.865 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T15:45:29.873 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T15:45:29.882 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-24T15:45:29.882 DEBUG:teuthology.orchestra.run.smithi042:> sudo service auditd rotate 2024-04-24T15:45:29.961 INFO:teuthology.orchestra.run.smithi042.stdout:Rotating logs: 2024-04-24T15:45:29.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-04-24T15:45:30.029 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-04-24T15:45:30.031 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T15:45:30.031 DEBUG:teuthology.orchestra.run.smithi042:> /usr/sbin/getenforce 2024-04-24T15:45:30.061 INFO:teuthology.orchestra.run.smithi042.stdout:Permissive 2024-04-24T15:45:30.061 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-04-24T15:45:30.084 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-04-24T15:45:30.084 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi042.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2024-04-24T15:45:30.084 DEBUG:teuthology.orchestra.run.smithi042:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T15:45:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:45:30.132 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T15:45:30.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T15:45:30.160 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T15:45:30.160 DEBUG:teuthology.orchestra.run.smithi042:> sudo /usr/sbin/setenforce permissive 2024-04-24T15:45:30.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-04-24T15:45:30.230 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T15:45:30.240 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T15:45:30.256 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T15:45:30.258 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi042.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T15:56:03.107 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi042.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-04-24T15:56:03.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-04-24T15:56:03.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T15:56:03.188 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-04-24T15:56:03.263 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-04-24T15:56:03.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-24T15:56:03.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T15:56:03.340 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-04-24T15:56:03.407 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-24T15:56:03.407 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T15:56:03.417 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T15:56:03.417 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T15:56:03.417 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T15:56:03.419 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T15:56:03.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T15:56:03.458 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T15:56:03.476 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T15:56:03.481 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T15:56:03.496 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T15:56:03.554 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: ntpd: command not found 2024-04-24T15:56:03.569 INFO:teuthology.orchestra.run.smithi042.stdout:506 Cannot talk to daemon 2024-04-24T15:56:03.585 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-04-24T15:56:03.586 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T15:56:03.597 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-04-24T15:56:03.603 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T15:56:03.611 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T15:56:03.626 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T15:56:03.681 INFO:teuthology.orchestra.run.smithi042.stderr:bash: ntpq: command not found 2024-04-24T15:56:03.685 INFO:teuthology.orchestra.run.smithi042.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T15:56:03.685 INFO:teuthology.orchestra.run.smithi042.stdout:=============================================================================== 2024-04-24T15:56:03.685 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:56:03.685 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:56:03.685 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:56:03.685 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:56:03.701 INFO:teuthology.orchestra.run.smithi136.stderr:bash: ntpq: command not found 2024-04-24T15:56:03.705 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T15:56:03.705 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-04-24T15:56:03.705 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:56:03.705 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:56:03.705 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:56:03.705 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T15:56:03.706 INFO:teuthology.run_tasks:Running task install... 2024-04-24T15:56:03.714 DEBUG:teuthology.task.install:project ceph 2024-04-24T15:56:03.714 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}} 2024-04-24T15:56:03.714 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror', 'rbd-nbd'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'} 2024-04-24T15:56:03.715 INFO:teuthology.task.install:Using flavor: default 2024-04-24T15:56:03.720 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T15:56:03.720 INFO:teuthology.task.install:extra packages: ['rbd-mirror', 'rbd-nbd'] 2024-04-24T15:56:03.720 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror', 'rbd-nbd'], 'extra_system_packages': ['pv'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'tag': None, 'wait_for_package': False} 2024-04-24T15:56:03.721 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:56:03.723 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror', 'rbd-nbd'], 'extra_system_packages': ['pv'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'tag': None, 'wait_for_package': False} 2024-04-24T15:56:03.723 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:56:03.989 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/ 2024-04-24T15:56:03.989 INFO:teuthology.task.install.rpm:Package version is 17.2.7-915.g330067db 2024-04-24T15:56:04.005 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/ 2024-04-24T15:56:04.006 INFO:teuthology.task.install.rpm:Package version is 17.2.7-915.g330067db 2024-04-24T15:56:04.173 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T15:56:04.173 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:56:04.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T15:56:04.189 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuriw4-testing-20240423.151325-quincy/330067dbce47655d1f762ef14d546d2fde16a6a9/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T15:56:04.189 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:56:04.189 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T15:56:04.203 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror, rbd-nbd, pv, pv, pv on remote rpm x86_64 2024-04-24T15:56:04.203 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/330067dbce47655d1f762ef14d546d2fde16a6a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T15:56:04.224 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror, rbd-nbd, pv, pv, pv, pv on remote rpm x86_64 2024-04-24T15:56:04.225 DEBUG:teuthology.orchestra.run.smithi042:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/330067dbce47655d1f762ef14d546d2fde16a6a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T15:56:04.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T15:56:04.301 DEBUG:teuthology.orchestra.run.smithi042:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T15:56:04.350 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T15:56:04.389 DEBUG:teuthology.orchestra.run.smithi042:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T15:56:04.417 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-04-24T15:56:04.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-04-24T15:56:04.421 INFO:teuthology.orchestra.run.smithi042.stdout:check_obsoletes = 1 2024-04-24T15:56:04.423 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum clean all 2024-04-24T15:56:04.796 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:04.857 INFO:teuthology.orchestra.run.smithi042.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:06.155 INFO:teuthology.orchestra.run.smithi042.stdout:63 files removed 2024-04-24T15:56:06.209 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd rbd-mirror rbd-nbd pv pv pv pv 2024-04-24T15:56:06.251 INFO:teuthology.orchestra.run.smithi136.stdout:63 files removed 2024-04-24T15:56:06.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd rbd-mirror rbd-nbd pv pv pv pv 2024-04-24T15:56:06.594 INFO:teuthology.orchestra.run.smithi042.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:06.636 INFO:teuthology.orchestra.run.smithi136.stdout:Updating Subscription Management repositories. 2024-04-24T15:56:08.861 INFO:teuthology.orchestra.run.smithi042.stdout:ceph packages for x86_64 123 kB/s | 78 kB 00:00 2024-04-24T15:56:08.870 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 125 kB/s | 78 kB 00:00 2024-04-24T15:56:09.196 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-04-24T15:56:09.270 INFO:teuthology.orchestra.run.smithi042.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-04-24T15:56:09.471 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 5.8 kB/s | 1.4 kB 00:00 2024-04-24T15:56:09.647 INFO:teuthology.orchestra.run.smithi136.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-04-24T15:56:09.654 INFO:teuthology.orchestra.run.smithi042.stdout:ceph source packages 4.1 kB/s | 1.4 kB 00:00 2024-04-24T15:56:09.830 INFO:teuthology.orchestra.run.smithi042.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-04-24T15:56:10.797 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-24T15:56:12.472 INFO:teuthology.orchestra.run.smithi042.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 16 MB 00:02 2024-04-24T15:56:14.406 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 410 kB/s | 24 kB 00:00 2024-04-24T15:56:15.749 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 53 MB/s | 67 MB 00:01 2024-04-24T15:56:16.589 INFO:teuthology.orchestra.run.smithi042.stdout:lab-extras 500 kB/s | 24 kB 00:00 2024-04-24T15:56:17.299 INFO:teuthology.orchestra.run.smithi042.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 14 MB/s | 8.8 MB 00:00 2024-04-24T15:56:20.492 INFO:teuthology.orchestra.run.smithi042.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 45 MB/s | 67 MB 00:01 2024-04-24T15:56:26.699 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 47 MB/s | 60 MB 00:01 2024-04-24T15:56:32.952 INFO:teuthology.orchestra.run.smithi042.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 60 MB 00:01 2024-04-24T15:56:37.406 INFO:teuthology.orchestra.run.smithi136.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 17 MB/s | 8.8 MB 00:00 2024-04-24T15:56:44.452 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-24T15:56:44.454 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-24T15:56:44.657 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-04-24T15:56:44.667 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================================================= 2024-04-24T15:56:44.667 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-04-24T15:56:44.667 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================================================= 2024-04-24T15:56:44.667 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:17.2.7-915.g330067db.el8 ceph 6.4 k 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:17.2.7-915.g330067db.el8 ceph 5.3 M 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 876 k 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-915.g330067db.el8 ceph 161 k 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:17.2.7-915.g330067db.el8 ceph 1.6 M 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 123 k 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 3.4 M 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 7.4 M 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 47 k 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T15:56:44.668 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:17.2.7-915.g330067db.el8 ceph 53 M 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 267 k 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 82 k 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 25 k 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:17.2.7-915.g330067db.el8 ceph 798 k 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 130 k 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: pv x86_64 1.6.6-7.el8 epel 64 k 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:17.2.7-915.g330067db.el8 ceph 190 k 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:17.2.7-915.g330067db.el8 ceph 370 k 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:17.2.7-915.g330067db.el8 ceph 358 k 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:17.2.7-915.g330067db.el8 ceph 107 k 2024-04-24T15:56:44.669 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 91 k 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:17.2.7-915.g330067db.el8 ceph 175 k 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.6 M 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 24 M 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 24 k 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:17.2.7-915.g330067db.el8 ceph 2.3 M 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 240 k 2024-04-24T15:56:44.670 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:17.2.7-915.g330067db.el8 ceph 4.0 M 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 14 k 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:17.2.7-915.g330067db.el8 ceph 24 k 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:17.2.7-915.g330067db.el8 ceph 179 k 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-24T15:56:44.671 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:17.2.7-915.g330067db.el8 ceph 505 k 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:17.2.7-915.g330067db.el8 ceph 5.6 M 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-24T15:56:44.672 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:17.2.7-915.g330067db.el8 ceph 45 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 117 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-24T15:56:44.673 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-24T15:56:44.674 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-24T15:56:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-24T15:56:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================================================= 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout:Install 102 Packages 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-04-24T15:56:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:56:44.678 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 199 M 2024-04-24T15:56:44.678 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-04-24T15:56:45.015 INFO:teuthology.orchestra.run.smithi136.stdout:(1/104): ceph-17.2.7-915.g330067db.el8.x86_64.r 34 kB/s | 6.4 kB 00:00 2024-04-24T15:56:45.607 INFO:teuthology.orchestra.run.smithi136.stdout:(2/104): ceph-fuse-17.2.7-915.g330067db.el8.x86 1.4 MB/s | 876 kB 00:00 2024-04-24T15:56:45.707 INFO:teuthology.orchestra.run.smithi136.stdout:(3/104): ceph-immutable-object-cache-17.2.7-915 1.6 MB/s | 161 kB 00:00 2024-04-24T15:56:45.815 INFO:teuthology.orchestra.run.smithi136.stdout:(4/104): ceph-base-17.2.7-915.g330067db.el8.x86 5.4 MB/s | 5.3 MB 00:00 2024-04-24T15:56:45.974 INFO:teuthology.orchestra.run.smithi136.stdout:(5/104): ceph-mgr-17.2.7-915.g330067db.el8.x86_ 9.9 MB/s | 1.6 MB 00:00 2024-04-24T15:56:46.241 INFO:teuthology.orchestra.run.smithi136.stdout:(6/104): ceph-mon-17.2.7-915.g330067db.el8.x86_ 15 MB/s | 4.0 MB 00:00 2024-04-24T15:56:46.491 INFO:teuthology.orchestra.run.smithi136.stdout:(7/104): ceph-mds-17.2.7-915.g330067db.el8.x86_ 2.9 MB/s | 2.3 MB 00:00 2024-04-24T15:56:47.250 INFO:teuthology.orchestra.run.smithi136.stdout:(8/104): ceph-osd-17.2.7-915.g330067db.el8.x86_ 17 MB/s | 17 MB 00:01 2024-04-24T15:56:47.308 INFO:teuthology.orchestra.run.smithi136.stdout:(9/104): ceph-selinux-17.2.7-915.g330067db.el8. 418 kB/s | 24 kB 00:00 2024-04-24T15:56:49.316 INFO:teuthology.orchestra.run.smithi136.stdout:(10/104): ceph-radosgw-17.2.7-915.g330067db.el8 6.0 MB/s | 17 MB 00:02 2024-04-24T15:56:49.383 INFO:teuthology.orchestra.run.smithi136.stdout:(11/104): libcephfs-devel-17.2.7-915.g330067db. 373 kB/s | 25 kB 00:00 2024-04-24T15:56:49.542 INFO:teuthology.orchestra.run.smithi136.stdout:(12/104): libcephfs2-17.2.7-915.g330067db.el8.x 4.9 MB/s | 798 kB 00:00 2024-04-24T15:56:49.751 INFO:teuthology.orchestra.run.smithi136.stdout:(13/104): ceph-common-17.2.7-915.g330067db.el8. 4.8 MB/s | 24 MB 00:04 2024-04-24T15:56:49.801 INFO:teuthology.orchestra.run.smithi136.stdout:(14/104): libcephsqlite-17.2.7-915.g330067db.el 694 kB/s | 179 kB 00:00 2024-04-24T15:56:49.826 INFO:teuthology.orchestra.run.smithi136.stdout:(15/104): librados-devel-17.2.7-915.g330067db.e 1.7 MB/s | 130 kB 00:00 2024-04-24T15:56:49.902 INFO:teuthology.orchestra.run.smithi136.stdout:(16/104): libradosstriper1-17.2.7-915.g330067db 4.9 MB/s | 505 kB 00:00 2024-04-24T15:56:49.985 INFO:teuthology.orchestra.run.smithi136.stdout:(17/104): python3-ceph-argparse-17.2.7-915.g330 538 kB/s | 45 kB 00:00 2024-04-24T15:56:50.077 INFO:teuthology.orchestra.run.smithi136.stdout:(18/104): python3-ceph-common-17.2.7-915.g33006 1.2 MB/s | 117 kB 00:00 2024-04-24T15:56:50.152 INFO:teuthology.orchestra.run.smithi136.stdout:(19/104): python3-cephfs-17.2.7-915.g330067db.e 2.5 MB/s | 190 kB 00:00 2024-04-24T15:56:50.244 INFO:teuthology.orchestra.run.smithi136.stdout:(20/104): python3-rados-17.2.7-915.g330067db.el 3.9 MB/s | 370 kB 00:00 2024-04-24T15:56:50.313 INFO:teuthology.orchestra.run.smithi042.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-24T15:56:50.315 INFO:teuthology.orchestra.run.smithi042.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-24T15:56:50.344 INFO:teuthology.orchestra.run.smithi136.stdout:(21/104): python3-rbd-17.2.7-915.g330067db.el8. 3.5 MB/s | 358 kB 00:00 2024-04-24T15:56:50.436 INFO:teuthology.orchestra.run.smithi136.stdout:(22/104): python3-rgw-17.2.7-915.g330067db.el8. 1.1 MB/s | 107 kB 00:00 2024-04-24T15:56:50.552 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-04-24T15:56:50.563 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================================================= 2024-04-24T15:56:50.563 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-04-24T15:56:50.563 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================================================= 2024-04-24T15:56:50.563 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-04-24T15:56:50.563 INFO:teuthology.orchestra.run.smithi042.stdout: ceph x86_64 2:17.2.7-915.g330067db.el8 ceph 6.4 k 2024-04-24T15:56:50.563 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base x86_64 2:17.2.7-915.g330067db.el8 ceph 5.3 M 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 876 k 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-915.g330067db.el8 ceph 161 k 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr x86_64 2:17.2.7-915.g330067db.el8 ceph 1.6 M 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 123 k 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 3.4 M 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 7.4 M 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-rook noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 47 k 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-radosgw x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-test x86_64 2:17.2.7-915.g330067db.el8 ceph 53 M 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-volume noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 267 k 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: cephadm noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 82 k 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 25 k 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2 x86_64 2:17.2.7-915.g330067db.el8 ceph 798 k 2024-04-24T15:56:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: librados-devel x86_64 2:17.2.7-915.g330067db.el8 ceph 130 k 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: pv x86_64 1.6.6-7.el8 epel 64 k 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs x86_64 2:17.2.7-915.g330067db.el8 ceph 190 k 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados x86_64 2:17.2.7-915.g330067db.el8 ceph 370 k 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rbd x86_64 2:17.2.7-915.g330067db.el8 ceph 358 k 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rgw x86_64 2:17.2.7-915.g330067db.el8 ceph 107 k 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-fuse x86_64 2:17.2.7-915.g330067db.el8 ceph 91 k 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-mirror x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-nbd x86_64 2:17.2.7-915.g330067db.el8 ceph 175 k 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout:Upgrading: 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: librados2 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.5 M 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: librbd1 x86_64 2:17.2.7-915.g330067db.el8 ceph 3.6 M 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 24 M 2024-04-24T15:56:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-grafana-dashboards noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 24 k 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds x86_64 2:17.2.7-915.g330067db.el8 ceph 2.3 M 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 240 k 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon x86_64 2:17.2.7-915.g330067db.el8 ceph 4.0 M 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-osd x86_64 2:17.2.7-915.g330067db.el8 ceph 17 M 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-prometheus-alerts noarch 2:17.2.7-915.g330067db.el8 ceph-noarch 14 k 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-selinux x86_64 2:17.2.7-915.g330067db.el8 ceph 24 k 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: libcephsqlite x86_64 2:17.2.7-915.g330067db.el8 ceph 179 k 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-04-24T15:56:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: libradosstriper1 x86_64 2:17.2.7-915.g330067db.el8 ceph 505 k 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: librgw2 x86_64 2:17.2.7-915.g330067db.el8 ceph 5.6 M 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-24T15:56:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse x86_64 2:17.2.7-915.g330067db.el8 ceph 45 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common x86_64 2:17.2.7-915.g330067db.el8 ceph 117 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-04-24T15:56:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-04-24T15:56:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-24T15:56:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================================================= 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout:Install 102 Packages 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout:Upgrade 2 Packages 2024-04-24T15:56:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:56:50.572 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 199 M 2024-04-24T15:56:50.572 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-04-24T15:56:50.995 INFO:teuthology.orchestra.run.smithi136.stdout:(23/104): ceph-test-17.2.7-915.g330067db.el8.x8 14 MB/s | 53 MB 00:03 2024-04-24T15:56:51.020 INFO:teuthology.orchestra.run.smithi136.stdout:(24/104): rbd-fuse-17.2.7-915.g330067db.el8.x86 155 kB/s | 91 kB 00:00 2024-04-24T15:56:51.052 INFO:teuthology.orchestra.run.smithi042.stdout:(1/104): ceph-17.2.7-915.g330067db.el8.x86_64.r 48 kB/s | 6.4 kB 00:00 2024-04-24T15:56:51.095 INFO:teuthology.orchestra.run.smithi136.stdout:(25/104): librgw2-17.2.7-915.g330067db.el8.x86_ 4.4 MB/s | 5.6 MB 00:01 2024-04-24T15:56:51.121 INFO:teuthology.orchestra.run.smithi136.stdout:(26/104): rbd-nbd-17.2.7-915.g330067db.el8.x86_ 1.7 MB/s | 175 kB 00:00 2024-04-24T15:56:51.162 INFO:teuthology.orchestra.run.smithi136.stdout:(27/104): ceph-grafana-dashboards-17.2.7-915.g3 364 kB/s | 24 kB 00:00 2024-04-24T15:56:51.213 INFO:teuthology.orchestra.run.smithi136.stdout:(28/104): ceph-mgr-cephadm-17.2.7-915.g330067db 1.3 MB/s | 123 kB 00:00 2024-04-24T15:56:51.288 INFO:teuthology.orchestra.run.smithi136.stdout:(29/104): rbd-mirror-17.2.7-915.g330067db.el8.x 12 MB/s | 3.5 MB 00:00 2024-04-24T15:56:51.363 INFO:teuthology.orchestra.run.smithi136.stdout:(30/104): ceph-mgr-modules-core-17.2.7-915.g330 3.1 MB/s | 240 kB 00:00 2024-04-24T15:56:51.430 INFO:teuthology.orchestra.run.smithi136.stdout:(31/104): ceph-mgr-rook-17.2.7-915.g330067db.el 711 kB/s | 47 kB 00:00 2024-04-24T15:56:51.497 INFO:teuthology.orchestra.run.smithi136.stdout:(32/104): ceph-prometheus-alerts-17.2.7-915.g33 217 kB/s | 14 kB 00:00 2024-04-24T15:56:51.527 INFO:teuthology.orchestra.run.smithi042.stdout:(2/104): ceph-fuse-17.2.7-915.g330067db.el8.x86 1.8 MB/s | 876 kB 00:00 2024-04-24T15:56:51.564 INFO:teuthology.orchestra.run.smithi136.stdout:(33/104): ceph-volume-17.2.7-915.g330067db.el8. 3.9 MB/s | 267 kB 00:00 2024-04-24T15:56:51.639 INFO:teuthology.orchestra.run.smithi136.stdout:(34/104): cephadm-17.2.7-915.g330067db.el8.noar 1.1 MB/s | 82 kB 00:00 2024-04-24T15:56:51.644 INFO:teuthology.orchestra.run.smithi042.stdout:(3/104): ceph-immutable-object-cache-17.2.7-915 1.4 MB/s | 161 kB 00:00 2024-04-24T15:56:51.773 INFO:teuthology.orchestra.run.smithi136.stdout:(35/104): ceph-mgr-dashboard-17.2.7-915.g330067 5.5 MB/s | 3.4 MB 00:00 2024-04-24T15:56:51.981 INFO:teuthology.orchestra.run.smithi136.stdout:(36/104): ceph-mgr-diskprediction-local-17.2.7- 9.7 MB/s | 7.4 MB 00:00 2024-04-24T15:56:52.023 INFO:teuthology.orchestra.run.smithi136.stdout:(37/104): liboath-2.6.2-3.el8.x86_64.rpm 235 kB/s | 59 kB 00:00 2024-04-24T15:56:52.049 INFO:teuthology.orchestra.run.smithi136.stdout:(38/104): gperftools-libs-2.7-9.el8.x86_64.rpm 748 kB/s | 306 kB 00:00 2024-04-24T15:56:52.132 INFO:teuthology.orchestra.run.smithi136.stdout:(39/104): pv-1.6.6-7.el8.x86_64.rpm 587 kB/s | 64 kB 00:00 2024-04-24T15:56:52.169 INFO:teuthology.orchestra.run.smithi042.stdout:(4/104): ceph-mds-17.2.7-915.g330067db.el8.x86_ 4.4 MB/s | 2.3 MB 00:00 2024-04-24T15:56:52.224 INFO:teuthology.orchestra.run.smithi136.stdout:(40/104): python3-asyncssh-2.7.0-2.el8.noarch.r 2.7 MB/s | 476 kB 00:00 2024-04-24T15:56:52.249 INFO:teuthology.orchestra.run.smithi136.stdout:(41/104): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 373 kB/s | 44 kB 00:00 2024-04-24T15:56:52.261 INFO:teuthology.orchestra.run.smithi042.stdout:(5/104): ceph-base-17.2.7-915.g330067db.el8.x86 3.9 MB/s | 5.3 MB 00:01 2024-04-24T15:56:52.291 INFO:teuthology.orchestra.run.smithi136.stdout:(42/104): libunwind-1.3.1-3.el8.x86_64.rpm 242 kB/s | 75 kB 00:00 2024-04-24T15:56:52.316 INFO:teuthology.orchestra.run.smithi136.stdout:(43/104): python3-beautifulsoup4-4.6.3-2.el8.1. 2.0 MB/s | 185 kB 00:00 2024-04-24T15:56:52.342 INFO:teuthology.orchestra.run.smithi136.stdout:(44/104): python3-cachetools-3.1.1-4.el8.noarch 358 kB/s | 33 kB 00:00 2024-04-24T15:56:52.375 INFO:teuthology.orchestra.run.smithi136.stdout:(45/104): python3-certifi-2018.10.15-7.el8.noar 195 kB/s | 16 kB 00:00 2024-04-24T15:56:52.409 INFO:teuthology.orchestra.run.smithi136.stdout:(46/104): python3-cheroot-8.5.2-1.el8.noarch.rp 1.8 MB/s | 173 kB 00:00 2024-04-24T15:56:52.467 INFO:teuthology.orchestra.run.smithi136.stdout:(47/104): python3-google-auth-1.1.1-10.el8.noar 885 kB/s | 81 kB 00:00 2024-04-24T15:56:52.501 INFO:teuthology.orchestra.run.smithi136.stdout:(48/104): python3-jaraco-6.2-6.el8.noarch.rpm 120 kB/s | 11 kB 00:00 2024-04-24T15:56:52.527 INFO:teuthology.orchestra.run.smithi136.stdout:(49/104): python3-cherrypy-18.4.0-1.el8.noarch. 2.0 MB/s | 384 kB 00:00 2024-04-24T15:56:52.553 INFO:teuthology.orchestra.run.smithi042.stdout:(6/104): ceph-mgr-17.2.7-915.g330067db.el8.x86_ 4.1 MB/s | 1.6 MB 00:00 2024-04-24T15:56:52.560 INFO:teuthology.orchestra.run.smithi136.stdout:(50/104): python3-jaraco-functools-2.0-4.el8.no 190 kB/s | 18 kB 00:00 2024-04-24T15:56:52.635 INFO:teuthology.orchestra.run.smithi136.stdout:(51/104): python3-logutils-0.3.5-11.el8.noarch. 449 kB/s | 49 kB 00:00 2024-04-24T15:56:52.660 INFO:teuthology.orchestra.run.smithi136.stdout:(52/104): python3-more-itertools-7.2.0-3.el8.no 586 kB/s | 59 kB 00:00 2024-04-24T15:56:52.694 INFO:teuthology.orchestra.run.smithi136.stdout:(53/104): python3-kubernetes-11.0.0-6.el8.noarc 6.1 MB/s | 1.2 MB 00:00 2024-04-24T15:56:52.728 INFO:teuthology.orchestra.run.smithi136.stdout:(54/104): python3-natsort-7.1.1-2.el8.noarch.rp 655 kB/s | 60 kB 00:00 2024-04-24T15:56:52.778 INFO:teuthology.orchestra.run.smithi042.stdout:(7/104): ceph-mon-17.2.7-915.g330067db.el8.x86_ 7.8 MB/s | 4.0 MB 00:00 2024-04-24T15:56:52.786 INFO:teuthology.orchestra.run.smithi136.stdout:(55/104): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2024-04-24T15:56:52.812 INFO:teuthology.orchestra.run.smithi136.stdout:(56/104): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2024-04-24T15:56:52.837 INFO:teuthology.orchestra.run.smithi136.stdout:(57/104): python3-repoze-lru-0.7-6.el8.noarch.r 309 kB/s | 34 kB 00:00 2024-04-24T15:56:52.896 INFO:teuthology.orchestra.run.smithi136.stdout:(58/104): python3-routes-2.4.1-12.el8.noarch.rp 1.8 MB/s | 196 kB 00:00 2024-04-24T15:56:52.921 INFO:teuthology.orchestra.run.smithi136.stdout:(59/104): python3-rsa-4.9-2.el8.noarch.rpm 590 kB/s | 64 kB 00:00 2024-04-24T15:56:52.946 INFO:teuthology.orchestra.run.smithi136.stdout:(60/104): python3-simplegeneric-0.8.1-17.el8.no 172 kB/s | 19 kB 00:00 2024-04-24T15:56:52.988 INFO:teuthology.orchestra.run.smithi136.stdout:(61/104): python3-singledispatch-3.4.0.3-18.el8 260 kB/s | 24 kB 00:00 2024-04-24T15:56:53.038 INFO:teuthology.orchestra.run.smithi136.stdout:(62/104): python3-tempora-1.14.1-5.el8.noarch.r 249 kB/s | 29 kB 00:00 2024-04-24T15:56:53.063 INFO:teuthology.orchestra.run.smithi136.stdout:(63/104): python3-trustme-0.6.0-4.el8.noarch.rp 235 kB/s | 27 kB 00:00 2024-04-24T15:56:53.339 INFO:teuthology.orchestra.run.smithi136.stdout:(64/104): python3-waitress-1.4.3-1.el8.noarch.r 729 kB/s | 255 kB 00:00 2024-04-24T15:56:53.364 INFO:teuthology.orchestra.run.smithi136.stdout:(65/104): python3-websocket-client-0.56.0-5.el8 204 kB/s | 61 kB 00:00 2024-04-24T15:56:53.397 INFO:teuthology.orchestra.run.smithi136.stdout:(66/104): python3-webob-1.8.5-1.el8.1.noarch.rp 700 kB/s | 251 kB 00:00 2024-04-24T15:56:53.423 INFO:teuthology.orchestra.run.smithi136.stdout:(67/104): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2024-04-24T15:56:53.456 INFO:teuthology.orchestra.run.smithi136.stdout:(68/104): python3-zc-lockfile-2.0-2.el8.noarch. 249 kB/s | 23 kB 00:00 2024-04-24T15:56:53.481 INFO:teuthology.orchestra.run.smithi136.stdout:(69/104): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 70 kB 00:00 2024-04-24T15:56:53.607 INFO:teuthology.orchestra.run.smithi136.stdout:(70/104): thrift-0.13.0-2.el8.x86_64.rpm 8.3 MB/s | 1.7 MB 00:00 2024-04-24T15:56:53.699 INFO:teuthology.orchestra.run.smithi136.stdout:(71/104): python3-oauthlib-2.1.0-1.el8.noarch.r 716 kB/s | 155 kB 00:00 2024-04-24T15:56:53.791 INFO:teuthology.orchestra.run.smithi136.stdout:(72/104): python3-jwt-1.6.1-2.el8.noarch.rpm 127 kB/s | 43 kB 00:00 2024-04-24T15:56:53.816 INFO:teuthology.orchestra.run.smithi136.stdout:(73/104): python3-requests-oauthlib-1.0.0-1.el8 365 kB/s | 43 kB 00:00 2024-04-24T15:56:53.866 INFO:teuthology.orchestra.run.smithi136.stdout:(74/104): python3-pycparser-2.14-14.el8.noarch. 421 kB/s | 109 kB 00:00 2024-04-24T15:56:53.900 INFO:teuthology.orchestra.run.smithi136.stdout:(75/104): fuse-2.9.7-17.el8.x86_64.rpm 762 kB/s | 83 kB 00:00 2024-04-24T15:56:54.108 INFO:teuthology.orchestra.run.smithi136.stdout:(76/104): libgfortran-8.5.0-20.el8.x86_64.rpm 2.2 MB/s | 645 kB 00:00 2024-04-24T15:56:54.134 INFO:teuthology.orchestra.run.smithi136.stdout:(77/104): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 238 kB 00:00 2024-04-24T15:56:54.137 INFO:teuthology.orchestra.run.smithi042.stdout:(8/104): ceph-common-17.2.7-915.g330067db.el8.x 7.3 MB/s | 24 MB 00:03 2024-04-24T15:56:54.176 INFO:teuthology.orchestra.run.smithi136.stdout:(78/104): libquadmath-8.5.0-20.el8.x86_64.rpm 558 kB/s | 172 kB 00:00 2024-04-24T15:56:54.195 INFO:teuthology.orchestra.run.smithi042.stdout:(9/104): ceph-selinux-17.2.7-915.g330067db.el8. 420 kB/s | 24 kB 00:00 2024-04-24T15:56:54.218 INFO:teuthology.orchestra.run.smithi136.stdout:(79/104): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 432 kB/s | 47 kB 00:00 2024-04-24T15:56:54.326 INFO:teuthology.orchestra.run.smithi136.stdout:(80/104): python3-jsonpointer-1.10-11.el8.noarc 133 kB/s | 20 kB 00:00 2024-04-24T15:56:54.360 INFO:teuthology.orchestra.run.smithi136.stdout:(81/104): python3-cryptography-3.2.1-7.el8_9.x8 2.4 MB/s | 559 kB 00:00 2024-04-24T15:56:54.427 INFO:teuthology.orchestra.run.smithi136.stdout:(82/104): python3-prettytable-0.7.2-14.el8.noar 210 kB/s | 44 kB 00:00 2024-04-24T15:56:54.452 INFO:teuthology.orchestra.run.smithi136.stdout:(83/104): python3-pyasn1-modules-0.3.7-6.el8.no 877 kB/s | 110 kB 00:00 2024-04-24T15:56:54.477 INFO:teuthology.orchestra.run.smithi136.stdout:(84/104): python3-jsonpatch-1.21-2.el8.noarch.r 226 kB/s | 27 kB 00:00 2024-04-24T15:56:54.503 INFO:teuthology.orchestra.run.smithi136.stdout:(85/104): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.6 MB/s | 126 kB 00:00 2024-04-24T15:56:54.578 INFO:teuthology.orchestra.run.smithi136.stdout:(86/104): python3-markupsafe-0.23-19.el8.x86_64 387 kB/s | 39 kB 00:00 2024-04-24T15:56:54.629 INFO:teuthology.orchestra.run.smithi042.stdout:(10/104): ceph-radosgw-17.2.7-915.g330067db.el8 9.1 MB/s | 17 MB 00:01 2024-04-24T15:56:54.653 INFO:teuthology.orchestra.run.smithi136.stdout:(87/104): python3-werkzeug-0.12.2-4.el8.noarch. 2.2 MB/s | 457 kB 00:00 2024-04-24T15:56:54.695 INFO:teuthology.orchestra.run.smithi136.stdout:(88/104): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.3 MB/s | 259 kB 00:00 2024-04-24T15:56:54.704 INFO:teuthology.orchestra.run.smithi042.stdout:(11/104): libcephfs-devel-17.2.7-915.g330067db. 333 kB/s | 25 kB 00:00 2024-04-24T15:56:54.720 INFO:teuthology.orchestra.run.smithi136.stdout:(89/104): python3-pyOpenSSL-19.0.0-1.el8.noarch 728 kB/s | 103 kB 00:00 2024-04-24T15:56:54.880 INFO:teuthology.orchestra.run.smithi042.stdout:(12/104): ceph-osd-17.2.7-915.g330067db.el8.x86 7.4 MB/s | 17 MB 00:02 2024-04-24T15:56:54.887 INFO:teuthology.orchestra.run.smithi136.stdout:(90/104): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2024-04-24T15:56:54.912 INFO:teuthology.orchestra.run.smithi136.stdout:(91/104): python3-jinja2-2.10.1-3.el8.noarch.rp 2.4 MB/s | 538 kB 00:00 2024-04-24T15:56:54.913 INFO:teuthology.orchestra.run.smithi042.stdout:(13/104): libcephfs2-17.2.7-915.g330067db.el8.x 3.7 MB/s | 798 kB 00:00 2024-04-24T15:56:54.960 INFO:teuthology.orchestra.run.smithi042.stdout:(14/104): libcephsqlite-17.2.7-915.g330067db.el 2.2 MB/s | 179 kB 00:00 2024-04-24T15:56:54.997 INFO:teuthology.orchestra.run.smithi042.stdout:(15/104): librados-devel-17.2.7-915.g330067db.e 1.5 MB/s | 130 kB 00:00 2024-04-24T15:56:55.012 INFO:teuthology.orchestra.run.smithi136.stdout:(92/104): python3-numpy-1.14.3-10.el8.x86_64.rp 13 MB/s | 3.7 MB 00:00 2024-04-24T15:56:55.030 INFO:teuthology.orchestra.run.smithi042.stdout:(16/104): libradosstriper1-17.2.7-915.g330067db 7.1 MB/s | 505 kB 00:00 2024-04-24T15:56:55.038 INFO:teuthology.orchestra.run.smithi136.stdout:(93/104): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.8 MB/s | 225 kB 00:00 2024-04-24T15:56:55.063 INFO:teuthology.orchestra.run.smithi136.stdout:(94/104): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-04-24T15:56:55.106 INFO:teuthology.orchestra.run.smithi042.stdout:(17/104): python3-ceph-argparse-17.2.7-915.g330 596 kB/s | 45 kB 00:00 2024-04-24T15:56:55.130 INFO:teuthology.orchestra.run.smithi136.stdout:(95/104): librdkafka-0.11.4-3.el8.x86_64.rpm 2.9 MB/s | 354 kB 00:00 2024-04-24T15:56:55.173 INFO:teuthology.orchestra.run.smithi042.stdout:(18/104): python3-ceph-common-17.2.7-915.g33006 1.7 MB/s | 117 kB 00:00 2024-04-24T15:56:55.197 INFO:teuthology.orchestra.run.smithi136.stdout:(96/104): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-04-24T15:56:55.273 INFO:teuthology.orchestra.run.smithi042.stdout:(19/104): python3-cephfs-17.2.7-915.g330067db.e 1.9 MB/s | 190 kB 00:00 2024-04-24T15:56:55.298 INFO:teuthology.orchestra.run.smithi136.stdout:(97/104): openblas-threads-0.3.15-6.el8.x86_64. 20 MB/s | 5.2 MB 00:00 2024-04-24T15:56:55.322 INFO:teuthology.orchestra.run.smithi136.stdout:(98/104): jq-1.6-7.el8.x86_64.rpm 1.6 MB/s | 202 kB 00:00 2024-04-24T15:56:55.365 INFO:teuthology.orchestra.run.smithi042.stdout:(20/104): python3-rados-17.2.7-915.g330067db.el 4.0 MB/s | 370 kB 00:00 2024-04-24T15:56:55.414 INFO:teuthology.orchestra.run.smithi136.stdout:(99/104): openblas-0.3.15-6.el8.x86_64.rpm 18 MB/s | 5.0 MB 00:00 2024-04-24T15:56:55.431 INFO:teuthology.orchestra.run.smithi136.stdout:(100/104): python3-pytz-2017.2-11.el8.noarch.rp 406 kB/s | 54 kB 00:00 2024-04-24T15:56:55.457 INFO:teuthology.orchestra.run.smithi042.stdout:(21/104): python3-rbd-17.2.7-915.g330067db.el8. 3.8 MB/s | 358 kB 00:00 2024-04-24T15:56:55.574 INFO:teuthology.orchestra.run.smithi042.stdout:(22/104): librgw2-17.2.7-915.g330067db.el8.x86_ 9.7 MB/s | 5.6 MB 00:00 2024-04-24T15:56:55.581 INFO:teuthology.orchestra.run.smithi136.stdout:(101/104): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-04-24T15:56:55.624 INFO:teuthology.orchestra.run.smithi042.stdout:(23/104): python3-rgw-17.2.7-915.g330067db.el8. 639 kB/s | 107 kB 00:00 2024-04-24T15:56:55.649 INFO:teuthology.orchestra.run.smithi042.stdout:(24/104): rbd-fuse-17.2.7-915.g330067db.el8.x86 1.2 MB/s | 91 kB 00:00 2024-04-24T15:56:55.725 INFO:teuthology.orchestra.run.smithi042.stdout:(25/104): rbd-nbd-17.2.7-915.g330067db.el8.x86_ 2.3 MB/s | 175 kB 00:00 2024-04-24T15:56:55.732 INFO:teuthology.orchestra.run.smithi136.stdout:(102/104): python3-scipy-1.0.0-21.module+el8.9. 34 MB/s | 14 MB 00:00 2024-04-24T15:56:55.821 INFO:teuthology.orchestra.run.smithi042.stdout:(26/104): ceph-grafana-dashboards-17.2.7-915.g3 254 kB/s | 24 kB 00:00 2024-04-24T15:56:55.934 INFO:teuthology.orchestra.run.smithi042.stdout:(27/104): ceph-mgr-cephadm-17.2.7-915.g330067db 1.1 MB/s | 123 kB 00:00 2024-04-24T15:56:56.307 INFO:teuthology.orchestra.run.smithi136.stdout:(103/104): librados2-17.2.7-915.g330067db.el8.x 4.0 MB/s | 3.5 MB 00:00 2024-04-24T15:56:56.359 INFO:teuthology.orchestra.run.smithi042.stdout:(28/104): rbd-mirror-17.2.7-915.g330067db.el8.x 4.7 MB/s | 3.5 MB 00:00 2024-04-24T15:56:56.443 INFO:teuthology.orchestra.run.smithi042.stdout:(29/104): ceph-mgr-dashboard-17.2.7-915.g330067 6.6 MB/s | 3.4 MB 00:00 2024-04-24T15:56:56.510 INFO:teuthology.orchestra.run.smithi042.stdout:(30/104): ceph-mgr-modules-core-17.2.7-915.g330 3.5 MB/s | 240 kB 00:00 2024-04-24T15:56:56.515 INFO:teuthology.orchestra.run.smithi136.stdout:(104/104): librbd1-17.2.7-915.g330067db.el8.x86 3.8 MB/s | 3.6 MB 00:00 2024-04-24T15:56:56.518 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:56:56.518 INFO:teuthology.orchestra.run.smithi136.stdout:Total 17 MB/s | 199 MB 00:11 2024-04-24T15:56:56.578 INFO:teuthology.orchestra.run.smithi042.stdout:(31/104): ceph-mgr-rook-17.2.7-915.g330067db.el 714 kB/s | 47 kB 00:00 2024-04-24T15:56:56.635 INFO:teuthology.orchestra.run.smithi042.stdout:(32/104): ceph-prometheus-alerts-17.2.7-915.g33 248 kB/s | 14 kB 00:00 2024-04-24T15:56:56.711 INFO:teuthology.orchestra.run.smithi042.stdout:(33/104): ceph-volume-17.2.7-915.g330067db.el8. 3.5 MB/s | 267 kB 00:00 2024-04-24T15:56:56.777 INFO:teuthology.orchestra.run.smithi042.stdout:(34/104): cephadm-17.2.7-915.g330067db.el8.noar 1.2 MB/s | 82 kB 00:00 2024-04-24T15:56:56.903 INFO:teuthology.orchestra.run.smithi042.stdout:(35/104): gperftools-libs-2.7-9.el8.x86_64.rpm 2.4 MB/s | 306 kB 00:00 2024-04-24T15:56:56.978 INFO:teuthology.orchestra.run.smithi042.stdout:(36/104): liboath-2.6.2-3.el8.x86_64.rpm 785 kB/s | 59 kB 00:00 2024-04-24T15:56:57.020 INFO:teuthology.orchestra.run.smithi042.stdout:(37/104): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2024-04-24T15:56:57.060 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-04-24T15:56:57.062 INFO:teuthology.orchestra.run.smithi042.stdout:(38/104): pv-1.6.6-7.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-24T15:56:57.129 INFO:teuthology.orchestra.run.smithi042.stdout:(39/104): python3-asyncssh-2.7.0-2.el8.noarch.r 7.0 MB/s | 476 kB 00:00 2024-04-24T15:56:57.152 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-04-24T15:56:57.152 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-04-24T15:56:57.162 INFO:teuthology.orchestra.run.smithi042.stdout:(40/104): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.3 MB/s | 44 kB 00:00 2024-04-24T15:56:57.288 INFO:teuthology.orchestra.run.smithi042.stdout:(41/104): ceph-mgr-diskprediction-local-17.2.7- 8.0 MB/s | 7.4 MB 00:00 2024-04-24T15:56:57.305 INFO:teuthology.orchestra.run.smithi042.stdout:(42/104): python3-beautifulsoup4-4.6.3-2.el8.1. 1.3 MB/s | 185 kB 00:00 2024-04-24T15:56:57.339 INFO:teuthology.orchestra.run.smithi042.stdout:(43/104): python3-certifi-2018.10.15-7.el8.noar 487 kB/s | 16 kB 00:00 2024-04-24T15:56:57.355 INFO:teuthology.orchestra.run.smithi042.stdout:(44/104): python3-cachetools-3.1.1-4.el8.noarch 493 kB/s | 33 kB 00:00 2024-04-24T15:56:57.389 INFO:teuthology.orchestra.run.smithi042.stdout:(45/104): python3-cheroot-8.5.2-1.el8.noarch.rp 3.4 MB/s | 173 kB 00:00 2024-04-24T15:56:57.431 INFO:teuthology.orchestra.run.smithi042.stdout:(46/104): python3-google-auth-1.1.1-10.el8.noar 1.9 MB/s | 81 kB 00:00 2024-04-24T15:56:57.456 INFO:teuthology.orchestra.run.smithi042.stdout:(47/104): python3-cherrypy-18.4.0-1.el8.noarch. 3.7 MB/s | 384 kB 00:00 2024-04-24T15:56:57.482 INFO:teuthology.orchestra.run.smithi042.stdout:(48/104): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2024-04-24T15:56:57.507 INFO:teuthology.orchestra.run.smithi042.stdout:(49/104): python3-jaraco-functools-2.0-4.el8.no 350 kB/s | 18 kB 00:00 2024-04-24T15:56:57.649 INFO:teuthology.orchestra.run.smithi042.stdout:(50/104): python3-logutils-0.3.5-11.el8.noarch. 344 kB/s | 49 kB 00:00 2024-04-24T15:56:57.691 INFO:teuthology.orchestra.run.smithi042.stdout:(51/104): python3-kubernetes-11.0.0-6.el8.noarc 5.6 MB/s | 1.2 MB 00:00 2024-04-24T15:56:57.716 INFO:teuthology.orchestra.run.smithi042.stdout:(52/104): python3-more-itertools-7.2.0-3.el8.no 879 kB/s | 59 kB 00:00 2024-04-24T15:56:57.742 INFO:teuthology.orchestra.run.smithi042.stdout:(53/104): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2024-04-24T15:56:57.767 INFO:teuthology.orchestra.run.smithi042.stdout:(54/104): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2024-04-24T15:56:57.792 INFO:teuthology.orchestra.run.smithi042.stdout:(55/104): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2024-04-24T15:56:57.843 INFO:teuthology.orchestra.run.smithi042.stdout:(56/104): python3-repoze-lru-0.7-6.el8.noarch.r 447 kB/s | 34 kB 00:00 2024-04-24T15:56:57.868 INFO:teuthology.orchestra.run.smithi042.stdout:(57/104): python3-routes-2.4.1-12.el8.noarch.rp 2.6 MB/s | 196 kB 00:00 2024-04-24T15:56:57.893 INFO:teuthology.orchestra.run.smithi042.stdout:(58/104): python3-rsa-4.9-2.el8.noarch.rpm 1.2 MB/s | 64 kB 00:00 2024-04-24T15:56:57.918 INFO:teuthology.orchestra.run.smithi042.stdout:(59/104): python3-simplegeneric-0.8.1-17.el8.no 374 kB/s | 19 kB 00:00 2024-04-24T15:56:57.944 INFO:teuthology.orchestra.run.smithi042.stdout:(60/104): python3-singledispatch-3.4.0.3-18.el8 477 kB/s | 24 kB 00:00 2024-04-24T15:56:57.969 INFO:teuthology.orchestra.run.smithi042.stdout:(61/104): python3-tempora-1.14.1-5.el8.noarch.r 582 kB/s | 29 kB 00:00 2024-04-24T15:56:57.994 INFO:teuthology.orchestra.run.smithi042.stdout:(62/104): python3-trustme-0.6.0-4.el8.noarch.rp 548 kB/s | 27 kB 00:00 2024-04-24T15:56:58.028 INFO:teuthology.orchestra.run.smithi042.stdout:(63/104): python3-waitress-1.4.3-1.el8.noarch.r 4.3 MB/s | 255 kB 00:00 2024-04-24T15:56:58.053 INFO:teuthology.orchestra.run.smithi042.stdout:(64/104): python3-webob-1.8.5-1.el8.1.noarch.rp 4.2 MB/s | 251 kB 00:00 2024-04-24T15:56:58.079 INFO:teuthology.orchestra.run.smithi042.stdout:(65/104): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2024-04-24T15:56:58.129 INFO:teuthology.orchestra.run.smithi042.stdout:(66/104): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2024-04-24T15:56:58.153 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-04-24T15:56:58.154 INFO:teuthology.orchestra.run.smithi042.stdout:(67/104): python3-zc-lockfile-2.0-2.el8.noarch. 305 kB/s | 23 kB 00:00 2024-04-24T15:56:58.162 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-04-24T15:56:58.680 INFO:teuthology.orchestra.run.smithi042.stdout:(68/104): ceph-test-17.2.7-915.g330067db.el8.x8 12 MB/s | 53 MB 00:04 2024-04-24T15:56:58.705 INFO:teuthology.orchestra.run.smithi042.stdout:(69/104): xmlstarlet-1.6.1-20.el8.x86_64.rpm 126 kB/s | 70 kB 00:00 2024-04-24T15:56:58.755 INFO:teuthology.orchestra.run.smithi042.stdout:(70/104): thrift-0.13.0-2.el8.x86_64.rpm 2.8 MB/s | 1.7 MB 00:00 2024-04-24T15:56:58.947 INFO:teuthology.orchestra.run.smithi042.stdout:(71/104): python3-jwt-1.6.1-2.el8.noarch.rpm 160 kB/s | 43 kB 00:00 2024-04-24T15:56:59.022 INFO:teuthology.orchestra.run.smithi042.stdout:(72/104): python3-oauthlib-2.1.0-1.el8.noarch.r 489 kB/s | 155 kB 00:00 2024-04-24T15:56:59.089 INFO:teuthology.orchestra.run.smithi042.stdout:(73/104): python3-requests-oauthlib-1.0.0-1.el8 301 kB/s | 43 kB 00:00 2024-04-24T15:56:59.115 INFO:teuthology.orchestra.run.smithi042.stdout:(74/104): python3-pycparser-2.14-14.el8.noarch. 319 kB/s | 109 kB 00:00 2024-04-24T15:56:59.198 INFO:teuthology.orchestra.run.smithi042.stdout:(75/104): fuse-2.9.7-17.el8.x86_64.rpm 472 kB/s | 83 kB 00:00 2024-04-24T15:56:59.249 INFO:teuthology.orchestra.run.smithi042.stdout:(76/104): libgfortran-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 645 kB 00:00 2024-04-24T15:56:59.282 INFO:teuthology.orchestra.run.smithi042.stdout:(77/104): libquadmath-8.5.0-20.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2024-04-24T15:56:59.357 INFO:teuthology.orchestra.run.smithi042.stdout:(78/104): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2024-04-24T15:56:59.374 INFO:teuthology.orchestra.run.smithi042.stdout:(79/104): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 375 kB/s | 47 kB 00:00 2024-04-24T15:56:59.474 INFO:teuthology.orchestra.run.smithi042.stdout:(80/104): python3-cryptography-3.2.1-7.el8_9.x8 2.9 MB/s | 559 kB 00:00 2024-04-24T15:56:59.500 INFO:teuthology.orchestra.run.smithi042.stdout:(81/104): python3-jsonpointer-1.10-11.el8.noarc 141 kB/s | 20 kB 00:00 2024-04-24T15:56:59.517 INFO:teuthology.orchestra.run.smithi042.stdout:(82/104): python3-prettytable-0.7.2-14.el8.noar 309 kB/s | 44 kB 00:00 2024-04-24T15:56:59.617 INFO:teuthology.orchestra.run.smithi042.stdout:(83/104): python3-pyasn1-modules-0.3.7-6.el8.no 769 kB/s | 110 kB 00:00 2024-04-24T15:56:59.667 INFO:teuthology.orchestra.run.smithi042.stdout:(84/104): python3-jsonpatch-1.21-2.el8.noarch.r 159 kB/s | 27 kB 00:00 2024-04-24T15:56:59.693 INFO:teuthology.orchestra.run.smithi042.stdout:(85/104): python3-pyasn1-0.3.7-6.el8.noarch.rpm 720 kB/s | 126 kB 00:00 2024-04-24T15:56:59.726 INFO:teuthology.orchestra.run.smithi042.stdout:(86/104): python3-werkzeug-0.12.2-4.el8.noarch. 4.1 MB/s | 457 kB 00:00 2024-04-24T15:56:59.802 INFO:teuthology.orchestra.run.smithi042.stdout:(87/104): python3-markupsafe-0.23-19.el8.x86_64 290 kB/s | 39 kB 00:00 2024-04-24T15:56:59.860 INFO:teuthology.orchestra.run.smithi042.stdout:(88/104): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2024-04-24T15:56:59.877 INFO:teuthology.orchestra.run.smithi042.stdout:(89/104): python3-pyOpenSSL-19.0.0-1.el8.noarch 688 kB/s | 103 kB 00:00 2024-04-24T15:57:00.050 INFO:teuthology.orchestra.run.smithi042.stdout:(90/104): python3-jinja2-2.10.1-3.el8.noarch.rp 2.8 MB/s | 538 kB 00:00 2024-04-24T15:57:00.128 INFO:teuthology.orchestra.run.smithi042.stdout:(91/104): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2024-04-24T15:57:00.178 INFO:teuthology.orchestra.run.smithi042.stdout:(92/104): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2024-04-24T15:57:00.262 INFO:teuthology.orchestra.run.smithi042.stdout:(93/104): python3-numpy-1.14.3-10.el8.x86_64.rp 9.7 MB/s | 3.7 MB 00:00 2024-04-24T15:57:00.304 INFO:teuthology.orchestra.run.smithi042.stdout:(94/104): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.3 MB/s | 225 kB 00:00 2024-04-24T15:57:00.362 INFO:teuthology.orchestra.run.smithi042.stdout:(95/104): librdkafka-0.11.4-3.el8.x86_64.rpm 1.9 MB/s | 354 kB 00:00 2024-04-24T15:57:00.421 INFO:teuthology.orchestra.run.smithi042.stdout:(96/104): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2024-04-24T15:57:00.521 INFO:teuthology.orchestra.run.smithi042.stdout:(97/104): openblas-threads-0.3.15-6.el8.x86_64. 20 MB/s | 5.2 MB 00:00 2024-04-24T15:57:00.571 INFO:teuthology.orchestra.run.smithi042.stdout:(98/104): jq-1.6-7.el8.x86_64.rpm 1.3 MB/s | 202 kB 00:00 2024-04-24T15:57:00.647 INFO:teuthology.orchestra.run.smithi042.stdout:(99/104): python3-pytz-2017.2-11.el8.noarch.rpm 433 kB/s | 54 kB 00:00 2024-04-24T15:57:00.722 INFO:teuthology.orchestra.run.smithi042.stdout:(100/104): openblas-0.3.15-6.el8.x86_64.rpm 14 MB/s | 5.0 MB 00:00 2024-04-24T15:57:00.797 INFO:teuthology.orchestra.run.smithi042.stdout:(101/104): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-04-24T15:57:01.039 INFO:teuthology.orchestra.run.smithi042.stdout:(102/104): python3-scipy-1.0.0-21.module+el8.9. 29 MB/s | 14 MB 00:00 2024-04-24T15:57:01.181 INFO:teuthology.orchestra.run.smithi042.stdout:(103/104): librados2-17.2.7-915.g330067db.el8.x 7.6 MB/s | 3.5 MB 00:00 2024-04-24T15:57:01.306 INFO:teuthology.orchestra.run.smithi042.stdout:(104/104): librbd1-17.2.7-915.g330067db.el8.x86 7.1 MB/s | 3.6 MB 00:00 2024-04-24T15:57:01.309 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-04-24T15:57:01.309 INFO:teuthology.orchestra.run.smithi042.stdout:Total 19 MB/s | 199 MB 00:10 2024-04-24T15:57:01.864 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-04-24T15:57:01.969 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-04-24T15:57:01.970 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-04-24T15:57:02.401 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-04-24T15:57:02.596 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/106 2024-04-24T15:57:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 2/106 2024-04-24T15:57:02.853 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 2/106 2024-04-24T15:57:03.015 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/106 2024-04-24T15:57:03.071 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-04-24T15:57:03.081 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-04-24T15:57:03.397 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/106 2024-04-24T15:57:03.590 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:17.2.7-915.g330067db.el8.x86_64 4/106 2024-04-24T15:57:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:17.2.7-915.g330067db.el8.x86_64 4/106 2024-04-24T15:57:04.191 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:17.2.7-915.g330067db.el8.x86_64 5/106 2024-04-24T15:57:04.322 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:17.2.7-915.g330067db.el8.x86_64 5/106 2024-04-24T15:57:04.548 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 6/106 2024-04-24T15:57:04.633 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 6/106 2024-04-24T15:57:04.802 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 7/106 2024-04-24T15:57:05.109 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 8/106 2024-04-24T15:57:05.785 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/106 2024-04-24T15:57:05.948 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/106 2024-04-24T15:57:06.054 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/106 2024-04-24T15:57:06.234 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/106 2024-04-24T15:57:06.405 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/106 2024-04-24T15:57:06.556 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 13/106 2024-04-24T15:57:06.664 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 13/106 2024-04-24T15:57:06.839 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_ 14/106 2024-04-24T15:57:06.918 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_ 14/106 2024-04-24T15:57:07.059 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 15/106 2024-04-24T15:57:07.315 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/106 2024-04-24T15:57:07.320 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-04-24T15:57:07.536 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/106 2024-04-24T15:57:07.554 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/106 2024-04-24T15:57:07.664 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 18/106 2024-04-24T15:57:07.719 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 2/106 2024-04-24T15:57:07.787 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 18/106 2024-04-24T15:57:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 2/106 2024-04-24T15:57:07.938 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 19/106 2024-04-24T15:57:08.003 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/106 2024-04-24T15:57:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/106 2024-04-24T15:57:08.552 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 19/106 2024-04-24T15:57:08.697 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 20/106 2024-04-24T15:57:08.711 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : librados2-2:17.2.7-915.g330067db.el8.x86_64 4/106 2024-04-24T15:57:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados2-2:17.2.7-915.g330067db.el8.x86_64 4/106 2024-04-24T15:57:09.301 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 20/106 2024-04-24T15:57:09.369 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : librbd1-2:17.2.7-915.g330067db.el8.x86_64 5/106 2024-04-24T15:57:09.656 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librbd1-2:17.2.7-915.g330067db.el8.x86_64 5/106 2024-04-24T15:57:09.869 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 6/106 2024-04-24T15:57:09.914 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.15-6.el8.x86_64 21/106 2024-04-24T15:57:09.973 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 6/106 2024-04-24T15:57:10.166 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 7/106 2024-04-24T15:57:10.490 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 8/106 2024-04-24T15:57:10.609 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 21/106 2024-04-24T15:57:10.895 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 22/106 2024-04-24T15:57:11.097 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 23/106 2024-04-24T15:57:11.245 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:17.2.7-915.g330067db.el8.x 24/106 2024-04-24T15:57:11.335 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/106 2024-04-24T15:57:11.357 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:17.2.7-915.g330067db.el8 25/106 2024-04-24T15:57:11.502 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/106 2024-04-24T15:57:11.609 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 26/106 2024-04-24T15:57:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librgw2-2:17.2.7-915.g330067db.el8.x86_64 10/106 2024-04-24T15:57:11.788 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 27/106 2024-04-24T15:57:11.975 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/106 2024-04-24T15:57:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/106 2024-04-24T15:57:12.335 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 13/106 2024-04-24T15:57:12.447 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 13/106 2024-04-24T15:57:12.602 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_ 14/106 2024-04-24T15:57:12.681 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_ 14/106 2024-04-24T15:57:12.876 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 15/106 2024-04-24T15:57:13.184 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/106 2024-04-24T15:57:13.409 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/106 2024-04-24T15:57:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 27/106 2024-04-24T15:57:13.906 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 28/106 2024-04-24T15:57:14.077 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 18/106 2024-04-24T15:57:14.226 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 18/106 2024-04-24T15:57:14.414 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 19/106 2024-04-24T15:57:14.864 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 29/106 2024-04-24T15:57:15.108 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 19/106 2024-04-24T15:57:15.260 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 20/106 2024-04-24T15:57:15.511 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 30/106 2024-04-24T15:57:15.746 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 31/106 2024-04-24T15:57:15.929 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 20/106 2024-04-24T15:57:15.965 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el8.noarch 32/106 2024-04-24T15:57:16.102 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : openblas-0.3.15-6.el8.x86_64 21/106 2024-04-24T15:57:16.174 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 33/106 2024-04-24T15:57:16.422 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 34/106 2024-04-24T15:57:16.641 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 35/106 2024-04-24T15:57:16.856 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 36/106 2024-04-24T15:57:16.857 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 21/106 2024-04-24T15:57:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 37/106 2024-04-24T15:57:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 37/106 2024-04-24T15:57:17.210 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 22/106 2024-04-24T15:57:17.324 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : jq-1.6-7.el8.x86_64 38/106 2024-04-24T15:57:17.461 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 23/106 2024-04-24T15:57:17.511 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 39/106 2024-04-24T15:57:17.618 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ceph-common-2:17.2.7-915.g330067db.el8.x 24/106 2024-04-24T15:57:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 40/106 2024-04-24T15:57:17.765 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ceph-argparse-2:17.2.7-915.g330067db.el8 25/106 2024-04-24T15:57:17.844 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 41/106 2024-04-24T15:57:18.027 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 42/106 2024-04-24T15:57:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 26/106 2024-04-24T15:57:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 27/106 2024-04-24T15:57:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.7-17.el8.x86_64 43/106 2024-04-24T15:57:18.500 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 44/106 2024-04-24T15:57:18.898 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 45/106 2024-04-24T15:57:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 46/106 2024-04-24T15:57:19.409 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/106 2024-04-24T15:57:19.632 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 48/106 2024-04-24T15:57:19.849 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 49/106 2024-04-24T15:57:20.178 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 50/106 2024-04-24T15:57:20.235 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 27/106 2024-04-24T15:57:20.369 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 51/106 2024-04-24T15:57:20.520 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 52/106 2024-04-24T15:57:20.657 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 28/106 2024-04-24T15:57:20.779 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 53/106 2024-04-24T15:57:20.953 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 54/106 2024-04-24T15:57:21.115 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 55/106 2024-04-24T15:57:21.357 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 56/106 2024-04-24T15:57:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 57/106 2024-04-24T15:57:21.733 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 29/106 2024-04-24T15:57:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 58/106 2024-04-24T15:57:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 59/106 2024-04-24T15:57:22.057 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 30/106 2024-04-24T15:57:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 60/106 2024-04-24T15:57:22.292 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 31/106 2024-04-24T15:57:22.308 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 61/106 2024-04-24T15:57:22.481 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 62/106 2024-04-24T15:57:22.521 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rsa-4.9-2.el8.noarch 32/106 2024-04-24T15:57:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 63/106 2024-04-24T15:57:22.731 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 33/106 2024-04-24T15:57:22.842 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 64/106 2024-04-24T15:57:23.032 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 65/106 2024-04-24T15:57:23.098 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 34/106 2024-04-24T15:57:23.256 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 66/106 2024-04-24T15:57:23.324 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 35/106 2024-04-24T15:57:23.408 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 67/106 2024-04-24T15:57:23.565 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 36/106 2024-04-24T15:57:23.631 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-2.6-1.el8.noarch 68/106 2024-04-24T15:57:23.784 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 37/106 2024-04-24T15:57:23.856 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 37/106 2024-04-24T15:57:23.868 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 69/106 2024-04-24T15:57:24.051 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 70/106 2024-04-24T15:57:24.127 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : jq-1.6-7.el8.x86_64 38/106 2024-04-24T15:57:24.261 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 71/106 2024-04-24T15:57:24.377 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 39/106 2024-04-24T15:57:24.578 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 40/106 2024-04-24T15:57:24.793 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 41/106 2024-04-24T15:57:25.000 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 42/106 2024-04-24T15:57:25.270 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : fuse-2.9.7-17.el8.x86_64 43/106 2024-04-24T15:57:25.301 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 72/106 2024-04-24T15:57:25.487 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 44/106 2024-04-24T15:57:25.622 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 73/106 2024-04-24T15:57:25.826 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 74/106 2024-04-24T15:57:25.903 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 45/106 2024-04-24T15:57:26.127 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 75/106 2024-04-24T15:57:26.224 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 46/106 2024-04-24T15:57:26.369 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 76/106 2024-04-24T15:57:26.451 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/106 2024-04-24T15:57:26.559 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 77/106 2024-04-24T15:57:26.701 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 48/106 2024-04-24T15:57:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 78/106 2024-04-24T15:57:26.978 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 49/106 2024-04-24T15:57:27.303 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 50/106 2024-04-24T15:57:27.535 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 51/106 2024-04-24T15:57:27.703 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 52/106 2024-04-24T15:57:27.954 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 53/106 2024-04-24T15:57:28.152 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 54/106 2024-04-24T15:57:28.341 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 55/106 2024-04-24T15:57:28.605 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 56/106 2024-04-24T15:57:28.817 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 57/106 2024-04-24T15:57:29.014 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 58/106 2024-04-24T15:57:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 59/106 2024-04-24T15:57:29.414 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 60/106 2024-04-24T15:57:29.649 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 61/106 2024-04-24T15:57:29.863 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 62/106 2024-04-24T15:57:30.020 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 63/106 2024-04-24T15:57:30.240 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 64/106 2024-04-24T15:57:30.485 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 65/106 2024-04-24T15:57:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 79/106 2024-04-24T15:57:30.730 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 66/106 2024-04-24T15:57:30.801 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 79/106 2024-04-24T15:57:30.890 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 67/106 2024-04-24T15:57:31.162 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-portend-2.6-1.el8.noarch 68/106 2024-04-24T15:57:31.369 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 79/106 2024-04-24T15:57:31.392 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 69/106 2024-04-24T15:57:31.601 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 70/106 2024-04-24T15:57:31.651 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 80/106 2024-04-24T15:57:31.736 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 80/106 2024-04-24T15:57:31.838 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 71/106 2024-04-24T15:57:31.884 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 81/106 2024-04-24T15:57:33.006 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 72/106 2024-04-24T15:57:33.328 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 73/106 2024-04-24T15:57:33.580 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 74/106 2024-04-24T15:57:33.947 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 75/106 2024-04-24T15:57:34.260 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 76/106 2024-04-24T15:57:34.554 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 77/106 2024-04-24T15:57:34.767 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 78/106 2024-04-24T15:57:39.143 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 79/106 2024-04-24T15:57:39.408 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 79/106 2024-04-24T15:57:40.043 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 79/106 2024-04-24T15:57:40.329 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 80/106 2024-04-24T15:57:40.409 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 80/106 2024-04-24T15:57:40.582 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 81/106 2024-04-24T15:57:51.828 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 81/106 2024-04-24T15:57:51.828 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-04-24T15:57:51.828 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-04-24T15:57:51.828 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /dev 2024-04-24T15:57:51.828 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /run 2024-04-24T15:57:51.828 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-04-24T15:57:51.828 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-04-24T15:57:51.829 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-04-24T15:57:51.829 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-04-24T15:57:51.829 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /dev 2024-04-24T15:57:51.829 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:57:52.196 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 82/106 2024-04-24T15:57:53.706 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 82/106 2024-04-24T15:57:53.929 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 83/106 2024-04-24T15:57:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 83/106 2024-04-24T15:57:54.321 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 84/106 2024-04-24T15:57:54.717 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 84/106 2024-04-24T15:57:54.879 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 85/106 2024-04-24T15:57:54.896 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 85/106 2024-04-24T15:57:56.595 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 86/106 2024-04-24T15:57:56.748 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:17.2.7-915.g330067db.el8.noarch 86/106 2024-04-24T15:57:56.893 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el 87/106 2024-04-24T15:57:57.039 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-915.g330067db.e 88/106 2024-04-24T15:57:57.237 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noar 89/106 2024-04-24T15:57:58.216 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noar 89/106 2024-04-24T15:57:58.503 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.no 90/106 2024-04-24T15:57:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.no 90/106 2024-04-24T15:57:59.619 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-915.g3300 91/106 2024-04-24T15:57:59.711 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-915.g3300 91/106 2024-04-24T15:58:00.049 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8 92/106 2024-04-24T15:58:00.211 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 93/106 2024-04-24T15:58:00.247 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 93/106 2024-04-24T15:58:00.378 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 94/106 2024-04-24T15:58:00.389 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 94/106 2024-04-24T15:58:01.971 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:17.2.7-915.g330067db.el8.x86_64 95/106 2024-04-24T15:58:02.144 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 96/106 2024-04-24T15:58:02.237 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 96/106 2024-04-24T15:58:02.411 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-915.g330067 97/106 2024-04-24T15:58:02.757 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-915.g330067 97/106 2024-04-24T15:58:02.919 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 98/106 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 81/106 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /sys 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /proc 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /dev 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /run 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /mnt 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /var/tmp 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /home 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /tmp 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /dev 2024-04-24T15:58:03.021 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:03.557 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 82/106 2024-04-24T15:58:05.232 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 82/106 2024-04-24T15:58:05.593 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 83/106 2024-04-24T15:58:05.840 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 83/106 2024-04-24T15:58:06.054 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 84/106 2024-04-24T15:58:06.495 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 84/106 2024-04-24T15:58:06.654 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 85/106 2024-04-24T15:58:06.674 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 85/106 2024-04-24T15:58:07.881 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 98/106 2024-04-24T15:58:08.185 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 99/106 2024-04-24T15:58:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 100/106 2024-04-24T15:58:08.484 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_6 101/106 2024-04-24T15:58:08.642 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 102/106 2024-04-24T15:58:08.708 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: cephadm-2:17.2.7-915.g330067db.el8.noarch 86/106 2024-04-24T15:58:08.785 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 103/106 2024-04-24T15:58:08.871 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : cephadm-2:17.2.7-915.g330067db.el8.noarch 86/106 2024-04-24T15:58:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : pv-1.6.6-7.el8.x86_64 104/106 2024-04-24T15:58:08.961 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 105/106 2024-04-24T15:58:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el 87/106 2024-04-24T15:58:09.195 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 105/106 2024-04-24T15:58:09.195 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 106/106 2024-04-24T15:58:09.203 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-915.g330067db.e 88/106 2024-04-24T15:58:09.428 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noar 89/106 2024-04-24T15:58:09.939 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 106/106 2024-04-24T15:58:09.939 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:17.2.7-915.g330067db.el8.x86_64 1/106 2024-04-24T15:58:09.939 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 2/106 2024-04-24T15:58:09.939 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 3/106 2024-04-24T15:58:09.939 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 4/106 2024-04-24T15:58:09.939 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-915.g330067 5/106 2024-04-24T15:58:09.939 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 6/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 7/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 8/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 9/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 10/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 11/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 12/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_6 13/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 14/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 15/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 16/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_ 17/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:17.2.7-915.g330067db.el8.x86_64 18/106 2024-04-24T15:58:09.940 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:17.2.7-915.g330067db.el8 19/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:17.2.7-915.g330067db.el8.x 20/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 21/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 22/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 23/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 24/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 25/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 26/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 27/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-915.g330067db.e 28/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noar 29/106 2024-04-24T15:58:09.941 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.no 30/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-915.g3300 31/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8 32/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 33/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el 34/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 35/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:17.2.7-915.g330067db.el8.noarch 36/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 37/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 38/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 39/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : pv-1.6.6-7.el8.x86_64 40/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 41/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 42/106 2024-04-24T15:58:09.942 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 43/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 44/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 45/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 46/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 47/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 48/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 49/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 50/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 51/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 52/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 53/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 54/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 55/106 2024-04-24T15:58:09.943 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-2.6-1.el8.noarch 56/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 57/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 58/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 59/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 60/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 61/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 62/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 63/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 64/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 65/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 66/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 67/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 68/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 69/106 2024-04-24T15:58:09.944 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 71/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 72/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 73/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 75/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 76/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 77/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 78/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 79/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 80/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 81/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 82/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 83/106 2024-04-24T15:58:09.945 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 85/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 86/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 87/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 88/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 89/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 90/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 91/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 92/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 93/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 94/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 95/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 96/106 2024-04-24T15:58:09.946 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 97/106 2024-04-24T15:58:09.947 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 98/106 2024-04-24T15:58:09.947 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : jq-1.6-7.el8.x86_64 99/106 2024-04-24T15:58:09.947 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 100/106 2024-04-24T15:58:09.947 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 101/106 2024-04-24T15:58:09.947 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 102/106 2024-04-24T15:58:09.947 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:17.2.7-915.g330067db.el8.x86_64 103/106 2024-04-24T15:58:09.947 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 104/106 2024-04-24T15:58:09.947 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:17.2.7-915.g330067db.el8.x86_64 105/106 2024-04-24T15:58:10.509 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noar 89/106 2024-04-24T15:58:10.800 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.no 90/106 2024-04-24T15:58:11.873 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.no 90/106 2024-04-24T15:58:12.086 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-915.g3300 91/106 2024-04-24T15:58:12.196 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-915.g3300 91/106 2024-04-24T15:58:12.563 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8 92/106 2024-04-24T15:58:12.767 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 93/106 2024-04-24T15:58:12.810 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 93/106 2024-04-24T15:58:12.961 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 94/106 2024-04-24T15:58:12.976 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 94/106 2024-04-24T15:58:14.779 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-2:17.2.7-915.g330067db.el8.x86_64 95/106 2024-04-24T15:58:14.942 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 96/106 2024-04-24T15:58:15.039 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 96/106 2024-04-24T15:58:15.195 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-915.g330067 97/106 2024-04-24T15:58:15.581 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-915.g330067 97/106 2024-04-24T15:58:15.770 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 98/106 2024-04-24T15:58:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 98/106 2024-04-24T15:58:21.587 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 99/106 2024-04-24T15:58:21.732 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 100/106 2024-04-24T15:58:21.928 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_6 101/106 2024-04-24T15:58:22.104 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 102/106 2024-04-24T15:58:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 106/106 2024-04-24T15:58:22.144 INFO:teuthology.orchestra.run.smithi136.stdout:Installed products updated. 2024-04-24T15:58:22.279 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 103/106 2024-04-24T15:58:22.482 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : pv-1.6.6-7.el8.x86_64 104/106 2024-04-24T15:58:22.482 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 105/106 2024-04-24T15:58:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 105/106 2024-04-24T15:58:22.706 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 106/106 2024-04-24T15:58:22.744 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:22.744 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-04-24T15:58:22.744 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.744 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.744 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-04-24T15:58:22.744 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.744 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.744 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.744 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:22.745 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: fuse-2.9.7-17.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: jq-1.6-7.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-24T15:58:22.746 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: pv-1.6.6-7.el8.x86_64 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-24T15:58:22.747 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-24T15:58:22.748 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-24T15:58:22.749 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-2.6-1.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-24T15:58:22.750 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-24T15:58:22.751 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.752 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.752 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:22.752 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-24T15:58:22.752 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-24T15:58:22.752 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-24T15:58:22.752 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:22.752 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-04-24T15:58:22.937 DEBUG:teuthology.parallel:result is None 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 106/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-2:17.2.7-915.g330067db.el8.x86_64 1/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-base-2:17.2.7-915.g330067db.el8.x86_64 2/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-common-2:17.2.7-915.g330067db.el8.x86_64 3/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 4/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-915.g330067 5/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 6/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 7/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 8/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 9/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 10/106 2024-04-24T15:58:23.545 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 11/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-test-2:17.2.7-915.g330067db.el8.x86_64 12/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_6 13/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 14/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 15/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados-devel-2:17.2.7-915.g330067db.el8.x86_64 16/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libradosstriper1-2:17.2.7-915.g330067db.el8.x86_ 17/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librgw2-2:17.2.7-915.g330067db.el8.x86_64 18/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ceph-argparse-2:17.2.7-915.g330067db.el8 19/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ceph-common-2:17.2.7-915.g330067db.el8.x 20/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 21/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rados-2:17.2.7-915.g330067db.el8.x86_64 22/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 23/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 24/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 25/106 2024-04-24T15:58:23.546 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 26/106 2024-04-24T15:58:23.547 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 27/106 2024-04-24T15:58:23.547 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-915.g330067db.e 28/106 2024-04-24T15:58:23.547 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noar 29/106 2024-04-24T15:58:23.547 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.no 30/106 2024-04-24T15:58:23.547 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-915.g3300 31/106 2024-04-24T15:58:23.547 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8 32/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 33/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-915.g330067db.el 34/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-volume-2:17.2.7-915.g330067db.el8.noarch 35/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : cephadm-2:17.2.7-915.g330067db.el8.noarch 36/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 37/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 38/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 39/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : pv-1.6.6-7.el8.x86_64 40/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 41/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 42/106 2024-04-24T15:58:23.554 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 43/106 2024-04-24T15:58:23.555 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 44/106 2024-04-24T15:58:23.555 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 45/106 2024-04-24T15:58:23.555 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 46/106 2024-04-24T15:58:23.555 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 47/106 2024-04-24T15:58:23.555 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 48/106 2024-04-24T15:58:23.555 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 49/106 2024-04-24T15:58:23.555 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 50/106 2024-04-24T15:58:23.556 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 51/106 2024-04-24T15:58:23.556 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 52/106 2024-04-24T15:58:23.556 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 53/106 2024-04-24T15:58:23.556 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 54/106 2024-04-24T15:58:23.556 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 55/106 2024-04-24T15:58:23.556 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-portend-2.6-1.el8.noarch 56/106 2024-04-24T15:58:23.556 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 57/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 58/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 59/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 60/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 61/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 62/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 63/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 64/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 65/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 66/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 67/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 68/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 69/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 70/106 2024-04-24T15:58:23.557 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 71/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 72/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 73/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 75/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 76/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 77/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 78/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 79/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 80/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 81/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 82/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 83/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/106 2024-04-24T15:58:23.558 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 85/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 86/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 87/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 88/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 89/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 90/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 91/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 92/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 93/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 94/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 95/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 96/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 97/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 98/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : jq-1.6-7.el8.x86_64 99/106 2024-04-24T15:58:23.559 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 100/106 2024-04-24T15:58:23.560 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 101/106 2024-04-24T15:58:23.560 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 102/106 2024-04-24T15:58:23.560 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados2-2:17.2.7-915.g330067db.el8.x86_64 103/106 2024-04-24T15:58:23.560 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 104/106 2024-04-24T15:58:23.560 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librbd1-2:17.2.7-915.g330067db.el8.x86_64 105/106 2024-04-24T15:58:37.338 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 106/106 2024-04-24T15:58:37.338 INFO:teuthology.orchestra.run.smithi042.stdout:Installed products updated. 2024-04-24T15:58:38.904 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:38.904 INFO:teuthology.orchestra.run.smithi042.stdout:Upgraded: 2024-04-24T15:58:38.904 INFO:teuthology.orchestra.run.smithi042.stdout: librados2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.904 INFO:teuthology.orchestra.run.smithi042.stdout: librbd1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.904 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-grafana-dashboards-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-immutable-object-cache-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:38.905 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-rook-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-osd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-prometheus-alerts-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-radosgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-selinux-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-test-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-volume-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: cephadm-2:17.2.7-915.g330067db.el8.noarch 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: fuse-2.9.7-17.el8.x86_64 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: jq-1.6-7.el8.x86_64 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.906 INFO:teuthology.orchestra.run.smithi042.stdout: libcephsqlite-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: librados-devel-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: libradosstriper1-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: librgw2-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-0.3.15-6.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-04-24T15:58:38.907 INFO:teuthology.orchestra.run.smithi042.stdout: pv-1.6.6-7.el8.x86_64 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-24T15:58:38.908 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-24T15:58:38.909 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-portend-2.6-1.el8.noarch 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.910 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-24T15:58:38.911 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-24T15:58:38.911 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rgw-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.911 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-24T15:58:38.911 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-24T15:58:38.911 INFO:teuthology.orchestra.run.smithi042.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-04-24T15:58:38.911 INFO:teuthology.orchestra.run.smithi042.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-24T15:58:38.912 INFO:teuthology.orchestra.run.smithi042.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-24T15:58:38.912 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-24T15:58:38.912 INFO:teuthology.orchestra.run.smithi042.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-24T15:58:38.912 INFO:teuthology.orchestra.run.smithi042.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-24T15:58:38.913 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-24T15:58:38.913 INFO:teuthology.orchestra.run.smithi042.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-24T15:58:38.913 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-24T15:58:38.913 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-24T15:58:38.913 INFO:teuthology.orchestra.run.smithi042.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-24T15:58:38.914 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-fuse-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.914 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-mirror-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.914 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-nbd-2:17.2.7-915.g330067db.el8.x86_64 2024-04-24T15:58:38.914 INFO:teuthology.orchestra.run.smithi042.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-24T15:58:38.914 INFO:teuthology.orchestra.run.smithi042.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-24T15:58:38.915 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-24T15:58:38.915 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:38.915 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-04-24T15:58:39.113 DEBUG:teuthology.parallel:result is None 2024-04-24T15:58:39.114 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:58:39.341 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T15:58:39.371 INFO:teuthology.orchestra.run.smithi042.stdout:17.2.7-915.g330067db.el8 2024-04-24T15:58:39.372 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-915.g330067db.el8 2024-04-24T15:58:39.372 INFO:teuthology.task.install:The correct ceph version 17.2.7-915.g330067db is installed. 2024-04-24T15:58:39.374 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T15:58:39.519 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T15:58:39.547 INFO:teuthology.orchestra.run.smithi136.stdout:17.2.7-915.g330067db.el8 2024-04-24T15:58:39.547 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-915.g330067db.el8 2024-04-24T15:58:39.548 INFO:teuthology.task.install:The correct ceph version 17.2.7-915.g330067db is installed. 2024-04-24T15:58:39.549 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T15:58:39.549 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:39.550 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T15:58:39.602 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:39.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T15:58:39.651 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T15:58:39.651 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:39.652 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T15:58:39.687 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T15:58:39.762 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:39.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T15:58:39.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T15:58:39.863 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T15:58:39.863 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:39.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T15:58:39.898 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T15:58:39.973 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:39.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T15:58:40.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T15:58:40.072 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T15:58:40.072 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:40.072 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T15:58:40.107 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T15:58:40.183 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:40.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T15:58:40.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T15:58:40.284 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T15:58:40.378 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-24T15:58:40.378 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 777 /var/log/ceph 2024-04-24T15:58:40.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2024-04-24T15:58:40.413 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-24T15:58:40.413 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T15:58:40.456 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T15:58:40.487 INFO:tasks.ceph:Creating extra log directories... 2024-04-24T15:58:40.487 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T15:58:40.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T15:58:40.579 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-04-24T15:58:40.579 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T15:58:40.580 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592', 'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/compare-mirror-image-alternate-primary-nbd}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7671592', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}], 'name': 'yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9954, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'targets': {'smithi042.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOgMBWGhQgRrZqrK+UViDZS8uTGr6VLDef+vszWRP2HNdvqzjekW/QRf9O1n2zFUkO6rlFsgHaI+6BgdEuX4NRs=', 'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBohcSRsl/owVXyMLt+NRae4wF031BSohDnw1SyHhey5bNWqKav/KxJtJdv2MOtxEdgw8B9hOglR8UDq4eq9vg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror', 'rbd-nbd'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/compare_mirror_image_alternate_primary.sh']}, 'env': {'RBD_DEVICE_TYPE': 'nbd'}, 'timeout': '3h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_14:09:24', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T15:58:40.581 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-24T15:58:40.606 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-24T15:58:40.625 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T15:58:40.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T15:58:40.701 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:40.701 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T15:58:40.764 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T15:58:40.764 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_1 2024-04-24T15:58:40.826 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T15:58:40.826 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:58:40.827 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 6h/6d Inode: 180695 Links: 1 2024-04-24T15:58:40.827 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:58:40.827 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:58:40.827 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-04-24 15:58:23.204128132 +0000 2024-04-24T15:58:40.827 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-04-24 15:55:14.719893855 +0000 2024-04-24T15:58:40.827 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-04-24 15:55:14.719893855 +0000 2024-04-24T15:58:40.827 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-04-24T15:58:40.827 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T15:58:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-04-24T15:58:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-04-24T15:58:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000124026 s, 4.1 MB/s 2024-04-24T15:58:40.901 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T15:58:40.965 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_2 2024-04-24T15:58:41.028 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T15:58:41.028 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:58:41.028 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 6h/6d Inode: 178819 Links: 1 2024-04-24T15:58:41.028 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:58:41.028 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:58:41.028 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-04-24 15:58:23.204128132 +0000 2024-04-24T15:58:41.028 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-04-24 15:55:15.092891155 +0000 2024-04-24T15:58:41.028 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-04-24 15:55:15.092891155 +0000 2024-04-24T15:58:41.028 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-04-24T15:58:41.029 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T15:58:41.101 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-04-24T15:58:41.101 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-04-24T15:58:41.101 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000125306 s, 4.1 MB/s 2024-04-24T15:58:41.102 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T15:58:41.165 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_3 2024-04-24T15:58:41.227 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T15:58:41.227 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:58:41.227 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 6h/6d Inode: 176119 Links: 1 2024-04-24T15:58:41.227 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:58:41.227 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:58:41.227 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-04-24 15:58:23.204128132 +0000 2024-04-24T15:58:41.227 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-04-24 15:55:15.452888550 +0000 2024-04-24T15:58:41.227 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-04-24 15:55:15.452888550 +0000 2024-04-24T15:58:41.228 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-04-24T15:58:41.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T15:58:41.300 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-04-24T15:58:41.300 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-04-24T15:58:41.300 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000131356 s, 3.9 MB/s 2024-04-24T15:58:41.301 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T15:58:41.365 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_4 2024-04-24T15:58:41.427 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T15:58:41.427 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:58:41.427 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 6h/6d Inode: 183496 Links: 1 2024-04-24T15:58:41.427 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:58:41.427 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:58:41.427 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-04-24 15:58:23.204128132 +0000 2024-04-24T15:58:41.427 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-04-24 15:55:15.817885909 +0000 2024-04-24T15:58:41.427 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-04-24 15:55:15.817885909 +0000 2024-04-24T15:58:41.427 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-04-24T15:58:41.428 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T15:58:41.499 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-04-24T15:58:41.500 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-04-24T15:58:41.500 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000121885 s, 4.2 MB/s 2024-04-24T15:58:41.501 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T15:58:41.564 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:41.565 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T15:58:41.565 INFO:tasks.ceph:Generating config... 2024-04-24T15:58:41.566 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-24T15:58:41.566 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-24T15:58:41.566 INFO:tasks.ceph:[client] log to stderr = False 2024-04-24T15:58:41.566 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-24T15:58:41.566 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:41.566 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:41.566 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:41.566 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:41.566 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:41.567 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:41.567 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:41.567 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:41.567 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:41.567 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:41.567 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:41.567 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:41.567 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:41.567 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:41.567 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-24T15:58:41.567 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-24T15:58:41.567 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-24T15:58:41.567 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T15:58:41.567 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T15:58:41.568 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T15:58:41.568 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T15:58:41.568 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T15:58:41.568 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T15:58:41.569 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T15:58:41.569 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-04-24T15:58:41.569 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-04-24T15:58:41.678 INFO:teuthology.orchestra.run.smithi042.stdout:creating /etc/ceph/cluster1.keyring 2024-04-24T15:58:41.681 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-04-24T15:58:41.733 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-24T15:58:41.764 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.42')] 2024-04-24T15:58:41.765 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.42', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-24T15:58:41.766 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:41.766 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T15:58:41.825 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.42 --print /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T15:58:41.911 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T15:58:41.912 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: generated fsid b9870ca7-a3e3-48c6-875b-287e67643414 2024-04-24T15:58:41.912 INFO:teuthology.orchestra.run.smithi042.stdout:setting min_mon_release = octopus 2024-04-24T15:58:41.912 INFO:teuthology.orchestra.run.smithi042.stdout:epoch 0 2024-04-24T15:58:41.912 INFO:teuthology.orchestra.run.smithi042.stdout:fsid b9870ca7-a3e3-48c6-875b-287e67643414 2024-04-24T15:58:41.912 INFO:teuthology.orchestra.run.smithi042.stdout:last_changed 2024-04-24T15:58:41.914873+0000 2024-04-24T15:58:41.912 INFO:teuthology.orchestra.run.smithi042.stdout:created 2024-04-24T15:58:41.914873+0000 2024-04-24T15:58:41.912 INFO:teuthology.orchestra.run.smithi042.stdout:min_mon_release 15 (octopus) 2024-04-24T15:58:41.912 INFO:teuthology.orchestra.run.smithi042.stdout:election_strategy: 1 2024-04-24T15:58:41.912 INFO:teuthology.orchestra.run.smithi042.stdout:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-04-24T15:58:41.912 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-04-24T15:58:41.914 DEBUG:teuthology.orchestra.run.smithi042:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T15:58:41.934 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID b9870ca7-a3e3-48c6-875b-287e67643414... 2024-04-24T15:58:41.935 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-24T15:58:41.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2024-04-24T15:58:42.033 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2024-04-24T15:58:42.034 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2024-04-24T15:58:42.035 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = 172.21.15.42 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: mon client directed command retry = 5 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on pool no app = False 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject socket failures = 5000 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = b9870ca7-a3e3-48c6-875b-287e67643414 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2024-04-24T15:58:42.036 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.037 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore block size = 96636764160 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore compression algorithm = zlib 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore compression mode = aggressive 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore fsck on mount = True 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluefs = 1/20 2024-04-24T15:58:42.038 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluestore = 1/20 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: debug rocksdb = 4/10 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd full ratio = 0.9 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd nearfull ratio = 0.8 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: osd failsafe full ratio = 0.95 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: osd objectstore = bluestore 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-04-24T15:58:42.039 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2024-04-24T15:58:42.040 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:58:42.041 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: debug rbd = 20 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: debug rbd_mirror = 15 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: log to stderr = False 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: rbd default features = 125 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.0] 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.042 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.1] 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.2] 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.3] 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.4] 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.5] 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.043 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.6] 2024-04-24T15:58:42.044 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.044 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.044 INFO:teuthology.orchestra.run.smithi042.stdout:[global] 2024-04-24T15:58:42.044 INFO:teuthology.orchestra.run.smithi042.stdout: chdir = "" 2024-04-24T15:58:42.044 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T15:58:42.044 INFO:teuthology.orchestra.run.smithi042.stdout: auth supported = cephx 2024-04-24T15:58:42.044 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.044 INFO:teuthology.orchestra.run.smithi042.stdout: filestore xattr use omap = true 2024-04-24T15:58:42.044 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: mon clock drift allowed = 1.000 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: osd crush chooseleaf type = 0 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: auth debug = true 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: ms die on old message = true 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: ms die on bug = true 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: mon pg warn max object skew = 0 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.045 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default size = 2 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd allow primary affinity = true 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd allow pg remap = true 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on legacy crush tunables = false 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on crush straw calc version zero = false 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on no sortbitwise = false 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on osd down out interval zero = false 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on too few osds = false 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: mon_allow_pool_size_one = true 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T15:58:42.046 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: osd default data pool replay window = 5 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: mon allow pool delete = true 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: mon cluster log file level = debug 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: debug asserts on shutdown = true 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: mon health detail to clog = false 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: mon host = 172.21.15.42 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: mon client directed command retry = 5 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on pool no app = False 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: ms inject socket failures = 5000 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: fsid = b9870ca7-a3e3-48c6-875b-287e67643414 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.047 INFO:teuthology.orchestra.run.smithi042.stdout:[osd] 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: osd journal size = 100 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: osd scrub load threshold = 5.0 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: osd scrub max interval = 600 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: osd mclock profile = high_recovery_ops 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: osd recover clone overlap = true 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: osd recovery max chunk = 1048576 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug shutdown = true 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug op order = true 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug verify stray on activate = true 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.048 INFO:teuthology.orchestra.run.smithi042.stdout: osd open classes on start = true 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug pg log writeout = true 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: osd deep scrub update digest min age = 30 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: osd map max advance = 10 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: journal zero on create = true 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: filestore ondisk finisher threads = 3 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: filestore apply finisher threads = 3 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: bdev debug aio = true 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug misdirected ops = true 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore block size = 96636764160 2024-04-24T15:58:42.049 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore compression algorithm = zlib 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore compression mode = aggressive 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore fsck on mount = True 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluefs = 1/20 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluestore = 1/20 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: debug osd = 20 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: debug rocksdb = 4/10 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd full ratio = 0.9 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd nearfull ratio = 0.8 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: osd failsafe full ratio = 0.95 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: osd objectstore = bluestore 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr] 2024-04-24T15:58:42.050 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: debug mgr = 20 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 20 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 20 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: mgr/telemetry/nag = false 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout:[mon] 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 20 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: debug paxos = 20 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 20 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: mon data avail warn = 5 2024-04-24T15:58:42.051 INFO:teuthology.orchestra.run.smithi042.stdout: mon mgr mkfs grace = 240 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd reporter subtree level = osd 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd prime pg temp = true 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: # 1m isn't quite enough 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: mon_down_mkfs_grace = 2m 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.052 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_filestore_osds = false 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout:[client] 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: rgw cache enabled = true 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: rgw enable ops log = true 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: rgw enable usage log = true 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: debug rbd = 20 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: debug rbd_mirror = 15 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: log to stderr = False 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: rbd default features = 125 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout:[mon.a] 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.0] 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.053 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.1] 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.2] 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.3] 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.4] 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.054 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.5] 2024-04-24T15:58:42.055 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.055 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.055 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.6] 2024-04-24T15:58:42.055 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:58:42.055 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:58:42.056 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-04-24T15:58:42.056 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-04-24T15:58:42.120 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T15:58:42.120 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:42.120 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-04-24T15:58:42.141 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:42.141 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-04-24T15:58:42.202 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi042.front.sepia.ceph.com 2024-04-24T15:58:42.202 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:42.202 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-24T15:58:42.202 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-24T15:58:42.290 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:42.290 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-24T15:58:42.352 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T15:58:42.352 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:42.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-24T15:58:42.353 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-24T15:58:42.396 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:42.397 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-24T15:58:42.457 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T15:58:42.457 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T15:58:42.457 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-24T15:58:42.522 INFO:teuthology.orchestra.run.smithi042.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-24T15:58:42.525 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T15:58:42.525 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T15:58:42.525 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-04-24T15:58:42.573 INFO:teuthology.orchestra.run.smithi042.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-04-24T15:58:42.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-04-24T15:58:42.639 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-04-24T15:58:42.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-24T15:58:42.697 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-24T15:58:42.710 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-24T15:58:42.791 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-04-24T15:58:42.804 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-24T15:58:42.886 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-04-24T15:58:42.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-24T15:58:42.980 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-04-24T15:58:42.992 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-24T15:58:43.073 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-04-24T15:58:43.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-24T15:58:43.166 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-04-24T15:58:43.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-24T15:58:43.261 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-04-24T15:58:43.274 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T15:58:43.274 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T15:58:43.274 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-04-24T15:58:43.310 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:43.310 INFO:tasks.ceph:role: cluster1.osd.0 2024-04-24T15:58:43.310 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi042.front.sepia.ceph.com 2024-04-24T15:58:43.311 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T15:58:43.385 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:58:43.385 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:58:43.385 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:58:43.385 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:58:43.385 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:58:43.386 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:58:43.386 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:58:43.386 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:58:43.386 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:58:43.386 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:58:43.409 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-04-24T15:58:43.411 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-04-24T15:58:43.411 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-04-24T15:58:43.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-04-24T15:58:43.539 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-04-24T15:58:43.655 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:43.655 INFO:tasks.ceph:role: cluster1.osd.1 2024-04-24T15:58:43.655 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi042.front.sepia.ceph.com 2024-04-24T15:58:43.655 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T15:58:43.687 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:58:43.687 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:58:43.687 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:58:43.687 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:58:43.687 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:58:43.687 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:58:43.688 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:58:43.688 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:58:43.688 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:58:43.688 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:58:43.710 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-04-24T15:58:43.712 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-04-24T15:58:43.712 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-04-24T15:58:43.800 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-04-24T15:58:43.837 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-04-24T15:58:43.913 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:58:43.913 INFO:tasks.ceph:role: cluster1.osd.2 2024-04-24T15:58:43.913 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi042.front.sepia.ceph.com 2024-04-24T15:58:43.914 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T15:58:43.987 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:58:43.987 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:58:43.987 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:58:43.987 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:58:43.987 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:58:43.987 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:58:43.987 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:58:43.988 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:58:43.988 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:58:43.988 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:58:44.010 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-04-24T15:58:44.012 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-04-24T15:58:44.012 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-04-24T15:58:44.100 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-04-24T15:58:44.137 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T15:58:44.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:44.233+0000 7f0011068540 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-04-24T15:58:44.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:44.233+0000 7f0011068540 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-04-24T15:58:44.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:44.233+0000 7f0011068540 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-24T15:58:44.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:44.233+0000 7f0011068540 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-24T15:58:44.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:44.233+0000 7f0011068540 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-24T15:58:44.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:44.234+0000 7f0011068540 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-04-24T15:58:44.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:44.762+0000 7f0011068540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:45.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:45.527+0000 7f0011068540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:46.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:46.290+0000 7f0011068540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:47.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-04-24T15:58:47.119 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T15:58:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:47.216+0000 7f5a711e2540 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-04-24T15:58:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:47.216+0000 7f5a711e2540 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-04-24T15:58:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:47.216+0000 7f5a711e2540 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-24T15:58:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:47.216+0000 7f5a711e2540 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-24T15:58:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:47.216+0000 7f5a711e2540 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-24T15:58:47.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:47.217+0000 7f5a711e2540 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-04-24T15:58:47.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:47.748+0000 7f5a711e2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:48.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:48.510+0000 7f5a711e2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:49.275+0000 7f5a711e2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:50.071 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-04-24T15:58:50.105 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T15:58:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:50.203+0000 7f7983dbf540 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-04-24T15:58:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:50.203+0000 7f7983dbf540 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-04-24T15:58:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:50.203+0000 7f7983dbf540 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-24T15:58:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:50.203+0000 7f7983dbf540 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-24T15:58:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:50.203+0000 7f7983dbf540 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-24T15:58:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:50.203+0000 7f7983dbf540 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-04-24T15:58:50.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:50.727+0000 7f7983dbf540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:51.498+0000 7f7983dbf540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:52.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-04-24T15:58:52.321+0000 7f7983dbf540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:58:53.057 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-04-24T15:58:53.092 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T15:58:53.092 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:53.092 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-04-24T15:58:53.167 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:53.167 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-04-24T15:58:53.242 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:53.242 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-04-24T15:58:53.317 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:53.318 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-04-24T15:58:53.392 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:53.392 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-04-24T15:58:53.454 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:53.455 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-04-24T15:58:53.475 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:53.476 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-04-24T15:58:53.535 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:53.536 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-04-24T15:58:53.595 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:53.595 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-04-24T15:58:53.655 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:53.655 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-04-24T15:58:53.713 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:53.713 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-04-24T15:58:53.772 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:53.772 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-04-24T15:58:53.831 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:58:53.831 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-04-24T15:58:53.888 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T15:58:53.888 DEBUG:teuthology.orchestra.run.smithi042:> sudo tee -a /etc/ceph/cluster1.keyring 2024-04-24T15:58:53.921 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr.x] 2024-04-24T15:58:53.921 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAyLClmko9ZHxAAjaK2EYbm47o37Jh9o6VIvw== 2024-04-24T15:58:53.921 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.0] 2024-04-24T15:58:53.921 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQA0LClmF70DDhAAuQ931R2mgUq4KEuEuPl6zA== 2024-04-24T15:58:53.921 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.1] 2024-04-24T15:58:53.921 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQA3LClmJdb8DBAAaEobYuVo2DRajubOqhSRUQ== 2024-04-24T15:58:53.921 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.2] 2024-04-24T15:58:53.921 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQA6LClmwLAyDBAAqjHqonGcuN1Ya3927VbQiQ== 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout:[client.0] 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAyLClmvD5dIhAAyxiHkf6kpIToMw5AzPYhWg== 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror] 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAyLClm7dBNJhAAt0xi84n2TQ3OMDEQnkIsTg== 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.0] 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAyLClmC4TCKRAANFDmeLsyB7UIkaTSnSyb1w== 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.1] 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAyLClm+SxfLxAAkZMY1lRF9jUcuB31s8OaCg== 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.2] 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAyLClmeU7+NBAA7YxF+5U3i9TPJfhPsrBNOw== 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.3] 2024-04-24T15:58:53.922 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAyLClmdJ+XOhAAE3jeNi1Mku3iBY+XZMFhTw== 2024-04-24T15:58:53.923 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.4] 2024-04-24T15:58:53.923 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAzLClmws6KBBAAFxGyoNvQSZez7nAO2nXJAQ== 2024-04-24T15:58:53.923 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.5] 2024-04-24T15:58:53.923 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAzLClmEbMZChAAHym/AYBxI625foTpuZmUzA== 2024-04-24T15:58:53.923 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.6] 2024-04-24T15:58:53.923 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAzLClmG26+DxAAvO8ESDgl9lEE02AzoCjSwQ== 2024-04-24T15:58:53.924 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T15:58:54.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:54.085 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:54.143 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:58:54.199 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:54.258 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:54.317 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:54.375 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:54.434 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:54.493 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:54.552 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:54.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:54.670 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:58:54.728 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T15:58:54.728 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-04-24T15:58:54.758 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-04-24T15:58:54.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-04-24T15:58:54.895 DEBUG:teuthology.orchestra.run.smithi042:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-04-24T15:58:54.956 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-04-24T15:58:54.956 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-04-24T15:58:54.956 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-04-24T15:58:54.999 INFO:tasks.ceph.cluster1.mon.a:Started 2024-04-24T15:58:55.000 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-04-24T15:58:55.000 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-04-24T15:58:55.000 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-04-24T15:58:55.002 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-04-24T15:58:55.003 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T15:58:55.003 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster cluster1 osd crush tunables default 2024-04-24T15:58:55.237 INFO:teuthology.orchestra.run.smithi042.stderr:adjusted tunables profile to default 2024-04-24T15:58:55.249 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-04-24T15:58:55.249 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:55.249 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-04-24T15:58:55.279 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:55.279 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-04-24T15:58:55.349 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:58:55.350 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-04-24T15:58:55.419 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster cluster1 osd new 6fc36614-9a61-4a2a-889a-bf72224374da 0 2024-04-24T15:58:55.679 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:55.680+0000 7f8fd5b04180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T15:58:55.691 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-04-24T15:58:55.705 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster cluster1 osd new bca44732-0beb-4819-9928-ca629336d3d2 1 2024-04-24T15:58:55.774 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:55.775+0000 7f8fd5b04180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T15:58:55.858 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:55.860+0000 7f8fd5b04180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T15:58:55.935 INFO:teuthology.orchestra.run.smithi042.stdout:1 2024-04-24T15:58:55.937 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:55.938+0000 7f8fd5b04180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T15:58:55.949 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster cluster1 osd new 56f0f362-3f29-4db1-8086-a02243c2e017 2 2024-04-24T15:58:56.134 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:56.136+0000 7f8fd5b04180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T15:58:56.180 INFO:teuthology.orchestra.run.smithi042.stdout:2 2024-04-24T15:58:56.194 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-04-24T15:58:56.194 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-04-24T15:58:56.196 INFO:tasks.ceph.cluster1.osd.0:Started 2024-04-24T15:58:56.196 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-04-24T15:58:56.196 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-04-24T15:58:56.198 INFO:tasks.ceph.cluster1.osd.1:Started 2024-04-24T15:58:56.198 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-04-24T15:58:56.199 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-04-24T15:58:56.200 INFO:tasks.ceph.cluster1.osd.2:Started 2024-04-24T15:58:56.201 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T15:58:56.321 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:56.323+0000 7f8fd5b04180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T15:58:56.403 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:56.405+0000 7f8fd5b04180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T15:58:56.426 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:58:56.427 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":5,"fsid":"b9870ca7-a3e3-48c6-875b-287e67643414","created":"2024-04-24T15:58:55.135792+0000","modified":"2024-04-24T15:58:56.181720+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T15:58:56.181720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6fc36614-9a61-4a2a-889a-bf72224374da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bca44732-0beb-4819-9928-ca629336d3d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"56f0f362-3f29-4db1-8086-a02243c2e017","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:58:56.441 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-04-24T15:58:56.442 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T15:58:56.556 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:56.557+0000 7f8fd5b04180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T15:58:56.640 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:56.641+0000 7f8fd5b04180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T15:58:56.744 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-24T15:58:56.815 INFO:tasks.ceph.cluster1.osd.2.smithi042.stderr:2024-04-24T15:58:56.817+0000 7f28be07d540 -1 Falling back to public interface 2024-04-24T15:58:56.828 INFO:tasks.ceph.cluster1.osd.0.smithi042.stderr:2024-04-24T15:58:56.830+0000 7f079d22d540 -1 Falling back to public interface 2024-04-24T15:58:56.831 INFO:tasks.ceph.cluster1.osd.1.smithi042.stderr:2024-04-24T15:58:56.832+0000 7f2d48994540 -1 Falling back to public interface 2024-04-24T15:58:56.836 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:56.838+0000 7f8fd5b04180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T15:58:56.982 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:56.983+0000 7f8fd5b04180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T15:58:56.984 INFO:teuthology.misc.health.smithi042.stdout: 2024-04-24T15:58:56.984 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":5,"fsid":"b9870ca7-a3e3-48c6-875b-287e67643414","created":"2024-04-24T15:58:55.135792+0000","modified":"2024-04-24T15:58:56.181720+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T15:58:56.181720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6fc36614-9a61-4a2a-889a-bf72224374da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bca44732-0beb-4819-9928-ca629336d3d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"56f0f362-3f29-4db1-8086-a02243c2e017","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:58:56.997 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-24T15:58:57.418 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:57.419+0000 7f8fd5b04180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T15:58:57.657 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:57.659+0000 7f8fd5b04180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T15:58:57.915 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:57.916+0000 7f8fd5b04180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T15:58:58.003 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:58.004+0000 7f8fd5b04180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T15:58:58.765 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:58.766+0000 7f8fd5b04180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T15:58:58.856 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:58.857+0000 7f8fd5b04180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T15:58:59.005 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:59.007+0000 7f8fd5b04180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T15:58:59.700 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:59.701+0000 7f8fd5b04180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T15:58:59.878 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:58:59.879+0000 7f8fd5b04180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T15:59:00.063 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:59:00.065+0000 7f8fd5b04180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T15:59:00.320 INFO:tasks.ceph.cluster1.osd.2.smithi042.stderr:2024-04-24T15:59:00.322+0000 7f28be07d540 -1 osd.2 0 log_to_monitors true 2024-04-24T15:59:00.349 INFO:tasks.ceph.cluster1.osd.0.smithi042.stderr:2024-04-24T15:59:00.351+0000 7f079d22d540 -1 osd.0 0 log_to_monitors true 2024-04-24T15:59:00.366 INFO:tasks.ceph.cluster1.osd.1.smithi042.stderr:2024-04-24T15:59:00.367+0000 7f2d48994540 -1 osd.1 0 log_to_monitors true 2024-04-24T15:59:00.974 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:59:00.975+0000 7f8fd5b04180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T15:59:01.218 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:59:01.219+0000 7f8fd5b04180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T15:59:01.299 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:59:01.301+0000 7f8fd5b04180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T15:59:01.487 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:59:01.489+0000 7f8fd5b04180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T15:59:02.511 INFO:tasks.ceph.cluster1.mgr.x.smithi042.stderr:2024-04-24T15:59:02.513+0000 7f8f945ba700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-24T15:59:03.300 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-24T15:59:03.668 INFO:teuthology.misc.health.smithi042.stdout: 2024-04-24T15:59:03.668 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":8,"fsid":"b9870ca7-a3e3-48c6-875b-287e67643414","created":"2024-04-24T15:58:55.135792+0000","modified":"2024-04-24T15:59:02.411537+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T15:58:56.181720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6fc36614-9a61-4a2a-889a-bf72224374da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bca44732-0beb-4819-9928-ca629336d3d2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"56f0f362-3f29-4db1-8086-a02243c2e017","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:59:03.687 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-24T15:59:04.627 INFO:tasks.ceph.cluster1.osd.0.smithi042.stderr:2024-04-24T15:59:04.629+0000 7f079564b700 -1 osd.0 0 waiting for initial osdmap 2024-04-24T15:59:04.632 INFO:tasks.ceph.cluster1.osd.0.smithi042.stderr:2024-04-24T15:59:04.634+0000 7f078cc3f700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:59:04.714 INFO:tasks.ceph.cluster1.osd.1.smithi042.stderr:2024-04-24T15:59:04.716+0000 7f2d3f5af700 -1 osd.1 0 waiting for initial osdmap 2024-04-24T15:59:04.718 INFO:tasks.ceph.cluster1.osd.1.smithi042.stderr:2024-04-24T15:59:04.719+0000 7f2d383a6700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:59:05.728 INFO:tasks.ceph.cluster1.osd.2.smithi042.stderr:2024-04-24T15:59:05.729+0000 7f28b4c98700 -1 osd.2 0 waiting for initial osdmap 2024-04-24T15:59:05.731 INFO:tasks.ceph.cluster1.osd.2.smithi042.stderr:2024-04-24T15:59:05.733+0000 7f28ada8f700 -1 osd.2 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:59:09.990 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-24T15:59:10.339 INFO:teuthology.misc.health.smithi042.stdout: 2024-04-24T15:59:10.339 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":13,"fsid":"b9870ca7-a3e3-48c6-875b-287e67643414","created":"2024-04-24T15:58:55.135792+0000","modified":"2024-04-24T15:59:09.736761+0000","last_up_change":"2024-04-24T15:59:06.732126+0000","last_in_change":"2024-04-24T15:58:56.181720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T15:59:07.567611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"6fc36614-9a61-4a2a-889a-bf72224374da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":40015},{"type":"v1","addr":"172.21.15.42:6809","nonce":40015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":40015},{"type":"v1","addr":"172.21.15.42:6811","nonce":40015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":40015},{"type":"v1","addr":"172.21.15.42:6815","nonce":40015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":40015},{"type":"v1","addr":"172.21.15.42:6813","nonce":40015}]},"public_addr":"172.21.15.42:6809/40015","cluster_addr":"172.21.15.42:6811/40015","heartbeat_back_addr":"172.21.15.42:6815/40015","heartbeat_front_addr":"172.21.15.42:6813/40015","state":["exists","up"]},{"osd":1,"uuid":"bca44732-0beb-4819-9928-ca629336d3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":40016},{"type":"v1","addr":"172.21.15.42:6817","nonce":40016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":40016},{"type":"v1","addr":"172.21.15.42:6819","nonce":40016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":40016},{"type":"v1","addr":"172.21.15.42:6823","nonce":40016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":40016},{"type":"v1","addr":"172.21.15.42:6821","nonce":40016}]},"public_addr":"172.21.15.42:6817/40016","cluster_addr":"172.21.15.42:6819/40016","heartbeat_back_addr":"172.21.15.42:6823/40016","heartbeat_front_addr":"172.21.15.42:6821/40016","state":["exists","up"]},{"osd":2,"uuid":"56f0f362-3f29-4db1-8086-a02243c2e017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":40000},{"type":"v1","addr":"172.21.15.42:6801","nonce":40000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":40000},{"type":"v1","addr":"172.21.15.42:6803","nonce":40000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":40000},{"type":"v1","addr":"172.21.15.42:6807","nonce":40000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":40000},{"type":"v1","addr":"172.21.15.42:6805","nonce":40000}]},"public_addr":"172.21.15.42:6801/40000","cluster_addr":"172.21.15.42:6803/40000","heartbeat_back_addr":"172.21.15.42:6807/40000","heartbeat_front_addr":"172.21.15.42:6805/40000","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.362853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.366407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.313848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:59:10.354 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-24T15:59:10.354 INFO:tasks.ceph:Creating RBD pool 2024-04-24T15:59:10.354 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-04-24T15:59:11.347 INFO:teuthology.orchestra.run.smithi042.stderr:pool 'rbd' created 2024-04-24T15:59:11.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-24T15:59:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-24T15:59:12.367 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-04-24T15:59:12.367 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-04-24T15:59:12.367 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-04-24T15:59:12.368 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-04-24T15:59:12.368 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T15:59:12.783 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:12.797 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6824","nonce":39618},{"type":"v1","addr":"172.21.15.42:6825","nonce":39618}]},"active_addr":"172.21.15.42:6825/39618","active_change":"2024-04-24T15:59:01.503604+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":2938048112}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":4220072558}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":2440898248}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":2633486429}]}]} 2024-04-24T15:59:12.799 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-04-24T15:59:12.799 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-04-24T15:59:12.799 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T15:59:13.148 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:13.148 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":15,"fsid":"b9870ca7-a3e3-48c6-875b-287e67643414","created":"2024-04-24T15:58:55.135792+0000","modified":"2024-04-24T15:59:12.343393+0000","last_up_change":"2024-04-24T15:59:06.732126+0000","last_in_change":"2024-04-24T15:58:56.181720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T15:59:07.567611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T15:59:10.739509+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"6fc36614-9a61-4a2a-889a-bf72224374da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":40015},{"type":"v1","addr":"172.21.15.42:6809","nonce":40015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":40015},{"type":"v1","addr":"172.21.15.42:6811","nonce":40015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":40015},{"type":"v1","addr":"172.21.15.42:6815","nonce":40015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":40015},{"type":"v1","addr":"172.21.15.42:6813","nonce":40015}]},"public_addr":"172.21.15.42:6809/40015","cluster_addr":"172.21.15.42:6811/40015","heartbeat_back_addr":"172.21.15.42:6815/40015","heartbeat_front_addr":"172.21.15.42:6813/40015","state":["exists","up"]},{"osd":1,"uuid":"bca44732-0beb-4819-9928-ca629336d3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":40016},{"type":"v1","addr":"172.21.15.42:6817","nonce":40016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":40016},{"type":"v1","addr":"172.21.15.42:6819","nonce":40016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":40016},{"type":"v1","addr":"172.21.15.42:6823","nonce":40016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":40016},{"type":"v1","addr":"172.21.15.42:6821","nonce":40016}]},"public_addr":"172.21.15.42:6817/40016","cluster_addr":"172.21.15.42:6819/40016","heartbeat_back_addr":"172.21.15.42:6823/40016","heartbeat_front_addr":"172.21.15.42:6821/40016","state":["exists","up"]},{"osd":2,"uuid":"56f0f362-3f29-4db1-8086-a02243c2e017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":40000},{"type":"v1","addr":"172.21.15.42:6801","nonce":40000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":40000},{"type":"v1","addr":"172.21.15.42:6803","nonce":40000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":40000},{"type":"v1","addr":"172.21.15.42:6807","nonce":40000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":40000},{"type":"v1","addr":"172.21.15.42:6805","nonce":40000}]},"public_addr":"172.21.15.42:6801/40000","cluster_addr":"172.21.15.42:6803/40000","heartbeat_back_addr":"172.21.15.42:6807/40000","heartbeat_front_addr":"172.21.15.42:6805/40000","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.362853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.366407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.313848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:59:13.164 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-04-24T15:59:13.164 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T15:59:13.512 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:13.512 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":16,"fsid":"b9870ca7-a3e3-48c6-875b-287e67643414","created":"2024-04-24T15:58:55.135792+0000","modified":"2024-04-24T15:59:13.345747+0000","last_up_change":"2024-04-24T15:59:06.732126+0000","last_in_change":"2024-04-24T15:58:56.181720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T15:59:07.567611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T15:59:10.739509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"6fc36614-9a61-4a2a-889a-bf72224374da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":40015},{"type":"v1","addr":"172.21.15.42:6809","nonce":40015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":40015},{"type":"v1","addr":"172.21.15.42:6811","nonce":40015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":40015},{"type":"v1","addr":"172.21.15.42:6815","nonce":40015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":40015},{"type":"v1","addr":"172.21.15.42:6813","nonce":40015}]},"public_addr":"172.21.15.42:6809/40015","cluster_addr":"172.21.15.42:6811/40015","heartbeat_back_addr":"172.21.15.42:6815/40015","heartbeat_front_addr":"172.21.15.42:6813/40015","state":["exists","up"]},{"osd":1,"uuid":"bca44732-0beb-4819-9928-ca629336d3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":40016},{"type":"v1","addr":"172.21.15.42:6817","nonce":40016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":40016},{"type":"v1","addr":"172.21.15.42:6819","nonce":40016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":40016},{"type":"v1","addr":"172.21.15.42:6823","nonce":40016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":40016},{"type":"v1","addr":"172.21.15.42:6821","nonce":40016}]},"public_addr":"172.21.15.42:6817/40016","cluster_addr":"172.21.15.42:6819/40016","heartbeat_back_addr":"172.21.15.42:6823/40016","heartbeat_front_addr":"172.21.15.42:6821/40016","state":["exists","up"]},{"osd":2,"uuid":"56f0f362-3f29-4db1-8086-a02243c2e017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":40000},{"type":"v1","addr":"172.21.15.42:6801","nonce":40000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":40000},{"type":"v1","addr":"172.21.15.42:6803","nonce":40000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":40000},{"type":"v1","addr":"172.21.15.42:6807","nonce":40000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":40000},{"type":"v1","addr":"172.21.15.42:6805","nonce":40000}]},"public_addr":"172.21.15.42:6801/40000","cluster_addr":"172.21.15.42:6803/40000","heartbeat_back_addr":"172.21.15.42:6807/40000","heartbeat_front_addr":"172.21.15.42:6805/40000","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.362853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.366407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.313848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:59:13.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-04-24T15:59:13.528 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-04-24T15:59:13.528 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-04-24T15:59:13.707 INFO:teuthology.orchestra.run.smithi042.stdout:38654705668 2024-04-24T15:59:13.707 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-24T15:59:13.745 INFO:teuthology.orchestra.run.smithi042.stdout:38654705670 2024-04-24T15:59:13.745 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-24T15:59:13.746 INFO:teuthology.orchestra.run.smithi042.stdout:42949672965 2024-04-24T15:59:13.746 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-24T15:59:14.069 INFO:teuthology.orchestra.run.smithi042.stdout:38654705666 2024-04-24T15:59:14.085 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705666 for osd.1 2024-04-24T15:59:14.159 INFO:teuthology.orchestra.run.smithi042.stdout:42949672963 2024-04-24T15:59:14.164 INFO:teuthology.orchestra.run.smithi042.stdout:38654705668 2024-04-24T15:59:14.176 INFO:tasks.ceph.ceph_manager.cluster1:need seq 42949672965 got 42949672963 for osd.2 2024-04-24T15:59:14.181 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705670 got 38654705668 for osd.0 2024-04-24T15:59:15.085 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-24T15:59:15.177 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-24T15:59:15.182 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-24T15:59:15.461 INFO:teuthology.orchestra.run.smithi042.stdout:38654705667 2024-04-24T15:59:15.477 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705667 for osd.1 2024-04-24T15:59:15.544 INFO:teuthology.orchestra.run.smithi042.stdout:42949672964 2024-04-24T15:59:15.559 INFO:tasks.ceph.ceph_manager.cluster1:need seq 42949672965 got 42949672964 for osd.2 2024-04-24T15:59:15.583 INFO:teuthology.orchestra.run.smithi042.stdout:38654705670 2024-04-24T15:59:15.599 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705670 got 38654705670 for osd.0 2024-04-24T15:59:15.599 DEBUG:teuthology.parallel:result is None 2024-04-24T15:59:16.478 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-24T15:59:16.561 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-24T15:59:16.843 INFO:teuthology.orchestra.run.smithi042.stdout:38654705668 2024-04-24T15:59:16.859 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705668 for osd.1 2024-04-24T15:59:16.859 DEBUG:teuthology.parallel:result is None 2024-04-24T15:59:16.915 INFO:teuthology.orchestra.run.smithi042.stdout:42949672965 2024-04-24T15:59:16.930 INFO:tasks.ceph.ceph_manager.cluster1:need seq 42949672965 got 42949672965 for osd.2 2024-04-24T15:59:16.930 DEBUG:teuthology.parallel:result is None 2024-04-24T15:59:16.930 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-24T15:59:16.931 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T15:59:17.277 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:17.279 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-04-24T15:59:17.292 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-24T15:59:15.510402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":904896,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":904315,"kb_avail":282210624,"statfs":{"total":289910292480,"available":288983678976,"internally_reserved":0,"allocated":516096,"data_stored":1290360,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":926018568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.164333"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359288+0000","last_change":"2024-04-24T15:59:12.357325+0000","last_active":"2024-04-24T15:59:12.359288+0000","last_peered":"2024-04-24T15:59:12.359288+0000","last_clean":"2024-04-24T15:59:12.359288+0000","last_became_active":"2024-04-24T15:59:12.357008+0000","last_became_peered":"2024-04-24T15:59:12.357008+0000","last_unstale":"2024-04-24T15:59:12.359288+0000","last_undegraded":"2024-04-24T15:59:12.359288+0000","last_fullsized":"2024-04-24T15:59:12.359288+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:15:20.941230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359353+0000","last_change":"2024-04-24T15:59:12.357605+0000","last_active":"2024-04-24T15:59:12.359353+0000","last_peered":"2024-04-24T15:59:12.359353+0000","last_clean":"2024-04-24T15:59:12.359353+0000","last_became_active":"2024-04-24T15:59:12.357265+0000","last_became_peered":"2024-04-24T15:59:12.357265+0000","last_unstale":"2024-04-24T15:59:12.359353+0000","last_undegraded":"2024-04-24T15:59:12.359353+0000","last_fullsized":"2024-04-24T15:59:12.359353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:15:21.669877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359470+0000","last_change":"2024-04-24T15:59:12.357122+0000","last_active":"2024-04-24T15:59:12.359470+0000","last_peered":"2024-04-24T15:59:12.359470+0000","last_clean":"2024-04-24T15:59:12.359470+0000","last_became_active":"2024-04-24T15:59:12.356793+0000","last_became_peered":"2024-04-24T15:59:12.356793+0000","last_unstale":"2024-04-24T15:59:12.359470+0000","last_undegraded":"2024-04-24T15:59:12.359470+0000","last_fullsized":"2024-04-24T15:59:12.359470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:35:23.805820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359840+0000","last_change":"2024-04-24T15:59:12.358877+0000","last_active":"2024-04-24T15:59:12.359840+0000","last_peered":"2024-04-24T15:59:12.359840+0000","last_clean":"2024-04-24T15:59:12.359840+0000","last_became_active":"2024-04-24T15:59:12.358571+0000","last_became_peered":"2024-04-24T15:59:12.358571+0000","last_unstale":"2024-04-24T15:59:12.359840+0000","last_undegraded":"2024-04-24T15:59:12.359840+0000","last_fullsized":"2024-04-24T15:59:12.359840+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:14:23.463356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":139,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.354449+0000","last_change":"2024-04-24T15:59:09.075498+0000","last_active":"2024-04-24T15:59:12.354449+0000","last_peered":"2024-04-24T15:59:12.354449+0000","last_clean":"2024-04-24T15:59:12.354449+0000","last_became_active":"2024-04-24T15:59:09.074520+0000","last_became_peered":"2024-04-24T15:59:09.074520+0000","last_unstale":"2024-04-24T15:59:12.354449+0000","last_undegraded":"2024-04-24T15:59:12.354449+0000","last_fullsized":"2024-04-24T15:59:12.354449+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_clean_scrub_stamp":"2024-04-24T15:59:07.733245+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:20.668162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359179+0000","last_change":"2024-04-24T15:59:12.355833+0000","last_active":"2024-04-24T15:59:12.359179+0000","last_peered":"2024-04-24T15:59:12.359179+0000","last_clean":"2024-04-24T15:59:12.359179+0000","last_became_active":"2024-04-24T15:59:12.355529+0000","last_became_peered":"2024-04-24T15:59:12.355529+0000","last_unstale":"2024-04-24T15:59:12.359179+0000","last_undegraded":"2024-04-24T15:59:12.359179+0000","last_fullsized":"2024-04-24T15:59:12.359179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:45.777687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.358884+0000","last_change":"2024-04-24T15:59:12.357100+0000","last_active":"2024-04-24T15:59:12.358884+0000","last_peered":"2024-04-24T15:59:12.358884+0000","last_clean":"2024-04-24T15:59:12.358884+0000","last_became_active":"2024-04-24T15:59:12.356512+0000","last_became_peered":"2024-04-24T15:59:12.356512+0000","last_unstale":"2024-04-24T15:59:12.358884+0000","last_undegraded":"2024-04-24T15:59:12.358884+0000","last_fullsized":"2024-04-24T15:59:12.358884+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:40.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359077+0000","last_change":"2024-04-24T15:59:12.356273+0000","last_active":"2024-04-24T15:59:12.359077+0000","last_peered":"2024-04-24T15:59:12.359077+0000","last_clean":"2024-04-24T15:59:12.359077+0000","last_became_active":"2024-04-24T15:59:12.355976+0000","last_became_peered":"2024-04-24T15:59:12.355976+0000","last_unstale":"2024-04-24T15:59:12.359077+0000","last_undegraded":"2024-04-24T15:59:12.359077+0000","last_fullsized":"2024-04-24T15:59:12.359077+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:14:20.180867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359147+0000","last_change":"2024-04-24T15:59:12.355878+0000","last_active":"2024-04-24T15:59:12.359147+0000","last_peered":"2024-04-24T15:59:12.359147+0000","last_clean":"2024-04-24T15:59:12.359147+0000","last_became_active":"2024-04-24T15:59:12.355473+0000","last_became_peered":"2024-04-24T15:59:12.355473+0000","last_unstale":"2024-04-24T15:59:12.359147+0000","last_undegraded":"2024-04-24T15:59:12.359147+0000","last_fullsized":"2024-04-24T15:59:12.359147+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:37:23.943905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301580,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94070260,"statfs":{"total":96636764160,"available":96327946240,"internally_reserved":0,"allocated":131072,"data_stored":35386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301664,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94070176,"statfs":{"total":96636764160,"available":96327860224,"internally_reserved":0,"allocated":196608,"data_stored":629220,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301652,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94070188,"statfs":{"total":96636764160,"available":96327872512,"internally_reserved":0,"allocated":188416,"data_stored":625754,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T15:59:17.293 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T15:59:17.641 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:17.644 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-04-24T15:59:17.656 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-04-24T15:59:17.510732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":904888,"kb_used_data":512,"kb_used_omap":4,"kb_used_meta":904315,"kb_avail":282210632,"statfs":{"total":289910292480,"available":288983687168,"internally_reserved":0,"allocated":524288,"data_stored":1293826,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":926018568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.164663"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359288+0000","last_change":"2024-04-24T15:59:12.357325+0000","last_active":"2024-04-24T15:59:12.359288+0000","last_peered":"2024-04-24T15:59:12.359288+0000","last_clean":"2024-04-24T15:59:12.359288+0000","last_became_active":"2024-04-24T15:59:12.357008+0000","last_became_peered":"2024-04-24T15:59:12.357008+0000","last_unstale":"2024-04-24T15:59:12.359288+0000","last_undegraded":"2024-04-24T15:59:12.359288+0000","last_fullsized":"2024-04-24T15:59:12.359288+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:15:20.941230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359353+0000","last_change":"2024-04-24T15:59:12.357605+0000","last_active":"2024-04-24T15:59:12.359353+0000","last_peered":"2024-04-24T15:59:12.359353+0000","last_clean":"2024-04-24T15:59:12.359353+0000","last_became_active":"2024-04-24T15:59:12.357265+0000","last_became_peered":"2024-04-24T15:59:12.357265+0000","last_unstale":"2024-04-24T15:59:12.359353+0000","last_undegraded":"2024-04-24T15:59:12.359353+0000","last_fullsized":"2024-04-24T15:59:12.359353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:15:21.669877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359470+0000","last_change":"2024-04-24T15:59:12.357122+0000","last_active":"2024-04-24T15:59:12.359470+0000","last_peered":"2024-04-24T15:59:12.359470+0000","last_clean":"2024-04-24T15:59:12.359470+0000","last_became_active":"2024-04-24T15:59:12.356793+0000","last_became_peered":"2024-04-24T15:59:12.356793+0000","last_unstale":"2024-04-24T15:59:12.359470+0000","last_undegraded":"2024-04-24T15:59:12.359470+0000","last_fullsized":"2024-04-24T15:59:12.359470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:35:23.805820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359840+0000","last_change":"2024-04-24T15:59:12.358877+0000","last_active":"2024-04-24T15:59:12.359840+0000","last_peered":"2024-04-24T15:59:12.359840+0000","last_clean":"2024-04-24T15:59:12.359840+0000","last_became_active":"2024-04-24T15:59:12.358571+0000","last_became_peered":"2024-04-24T15:59:12.358571+0000","last_unstale":"2024-04-24T15:59:12.359840+0000","last_undegraded":"2024-04-24T15:59:12.359840+0000","last_fullsized":"2024-04-24T15:59:12.359840+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:14:23.463356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":139,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.354449+0000","last_change":"2024-04-24T15:59:09.075498+0000","last_active":"2024-04-24T15:59:12.354449+0000","last_peered":"2024-04-24T15:59:12.354449+0000","last_clean":"2024-04-24T15:59:12.354449+0000","last_became_active":"2024-04-24T15:59:09.074520+0000","last_became_peered":"2024-04-24T15:59:09.074520+0000","last_unstale":"2024-04-24T15:59:12.354449+0000","last_undegraded":"2024-04-24T15:59:12.354449+0000","last_fullsized":"2024-04-24T15:59:12.354449+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_clean_scrub_stamp":"2024-04-24T15:59:07.733245+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:20.668162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359179+0000","last_change":"2024-04-24T15:59:12.355833+0000","last_active":"2024-04-24T15:59:12.359179+0000","last_peered":"2024-04-24T15:59:12.359179+0000","last_clean":"2024-04-24T15:59:12.359179+0000","last_became_active":"2024-04-24T15:59:12.355529+0000","last_became_peered":"2024-04-24T15:59:12.355529+0000","last_unstale":"2024-04-24T15:59:12.359179+0000","last_undegraded":"2024-04-24T15:59:12.359179+0000","last_fullsized":"2024-04-24T15:59:12.359179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:45.777687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.358884+0000","last_change":"2024-04-24T15:59:12.357100+0000","last_active":"2024-04-24T15:59:12.358884+0000","last_peered":"2024-04-24T15:59:12.358884+0000","last_clean":"2024-04-24T15:59:12.358884+0000","last_became_active":"2024-04-24T15:59:12.356512+0000","last_became_peered":"2024-04-24T15:59:12.356512+0000","last_unstale":"2024-04-24T15:59:12.358884+0000","last_undegraded":"2024-04-24T15:59:12.358884+0000","last_fullsized":"2024-04-24T15:59:12.358884+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:40.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359077+0000","last_change":"2024-04-24T15:59:12.356273+0000","last_active":"2024-04-24T15:59:12.359077+0000","last_peered":"2024-04-24T15:59:12.359077+0000","last_clean":"2024-04-24T15:59:12.359077+0000","last_became_active":"2024-04-24T15:59:12.355976+0000","last_became_peered":"2024-04-24T15:59:12.355976+0000","last_unstale":"2024-04-24T15:59:12.359077+0000","last_undegraded":"2024-04-24T15:59:12.359077+0000","last_fullsized":"2024-04-24T15:59:12.359077+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:14:20.180867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:12.359147+0000","last_change":"2024-04-24T15:59:12.355878+0000","last_active":"2024-04-24T15:59:12.359147+0000","last_peered":"2024-04-24T15:59:12.359147+0000","last_clean":"2024-04-24T15:59:12.359147+0000","last_became_active":"2024-04-24T15:59:12.355473+0000","last_became_peered":"2024-04-24T15:59:12.355473+0000","last_unstale":"2024-04-24T15:59:12.359147+0000","last_undegraded":"2024-04-24T15:59:12.359147+0000","last_fullsized":"2024-04-24T15:59:12.359147+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:37:23.943905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301580,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94070260,"statfs":{"total":96636764160,"available":96327946240,"internally_reserved":0,"allocated":131072,"data_stored":35386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301664,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94070176,"statfs":{"total":96636764160,"available":96327860224,"internally_reserved":0,"allocated":196608,"data_stored":629220,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301644,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94070196,"statfs":{"total":96636764160,"available":96327880704,"internally_reserved":0,"allocated":196608,"data_stored":629220,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T15:59:17.657 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-24T15:59:17.657 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-04-24T15:59:17.657 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-04-24T15:59:17.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-04-24T15:59:18.054 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:18.054 INFO:teuthology.orchestra.run.smithi042.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T15:59:18.069 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-04-24T15:59:18.070 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T15:59:18.083 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-04-24T15:59:18.083 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T15:59:18.084 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592', 'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/compare-mirror-image-alternate-primary-nbd}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7671592', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}], 'name': 'yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9954, 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuriw4-testing-20240423.151325-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_330067dbce47655d1f762ef14d546d2fde16a6a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9', 'targets': {'smithi042.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOgMBWGhQgRrZqrK+UViDZS8uTGr6VLDef+vszWRP2HNdvqzjekW/QRf9O1n2zFUkO6rlFsgHaI+6BgdEuX4NRs=', 'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOBohcSRsl/owVXyMLt+NRae4wF031BSohDnw1SyHhey5bNWqKav/KxJtJdv2MOtxEdgw8B9hOglR8UDq4eq9vg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror', 'rbd-nbd'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '330067dbce47655d1f762ef14d546d2fde16a6a9'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/compare_mirror_image_alternate_primary.sh']}, 'env': {'RBD_DEVICE_TYPE': 'nbd'}, 'timeout': '3h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_14:09:24', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T15:59:18.084 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-24T15:59:18.087 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-24T15:59:18.107 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T15:59:18.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T15:59:18.179 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:18.179 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T15:59:18.239 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T15:59:18.239 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-04-24T15:59:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T15:59:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:59:18.299 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 176035 Links: 1 2024-04-24T15:59:18.299 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:59:18.299 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:59:18.299 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-24 15:58:09.619315966 +0000 2024-04-24T15:59:18.299 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-24 15:54:34.937052950 +0000 2024-04-24T15:59:18.299 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-24 15:54:34.937052950 +0000 2024-04-24T15:59:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-04-24T15:59:18.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T15:59:18.367 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-24T15:59:18.367 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-24T15:59:18.368 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000264332 s, 1.9 MB/s 2024-04-24T15:59:18.369 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T15:59:18.428 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-04-24T15:59:18.487 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T15:59:18.487 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:59:18.487 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 183377 Links: 1 2024-04-24T15:59:18.487 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:59:18.488 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:59:18.488 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-24 15:58:09.619315966 +0000 2024-04-24T15:59:18.488 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-24 15:54:35.288045119 +0000 2024-04-24T15:59:18.488 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-24 15:54:35.288045119 +0000 2024-04-24T15:59:18.488 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-04-24T15:59:18.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T15:59:18.555 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-24T15:59:18.556 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-24T15:59:18.556 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000193028 s, 2.7 MB/s 2024-04-24T15:59:18.557 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T15:59:18.615 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-04-24T15:59:18.674 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T15:59:18.674 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:59:18.674 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 185346 Links: 1 2024-04-24T15:59:18.674 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:59:18.674 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:59:18.674 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-24 15:58:09.619315966 +0000 2024-04-24T15:59:18.674 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-24 15:54:35.626037577 +0000 2024-04-24T15:59:18.674 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-24 15:54:35.626037577 +0000 2024-04-24T15:59:18.674 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-04-24T15:59:18.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T15:59:18.739 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-24T15:59:18.740 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-24T15:59:18.740 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000213044 s, 2.4 MB/s 2024-04-24T15:59:18.741 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T15:59:18.799 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-04-24T15:59:18.857 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T15:59:18.858 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T15:59:18.858 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 6h/6d Inode: 174004 Links: 1 2024-04-24T15:59:18.858 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T15:59:18.858 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T15:59:18.858 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-24 15:58:09.619315966 +0000 2024-04-24T15:59:18.859 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-24 15:54:35.957030192 +0000 2024-04-24T15:59:18.859 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-24 15:54:35.957030192 +0000 2024-04-24T15:59:18.859 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-04-24T15:59:18.859 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T15:59:18.924 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-24T15:59:18.924 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-24T15:59:18.924 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000261092 s, 2.0 MB/s 2024-04-24T15:59:18.925 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T15:59:18.983 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:59:18.984 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T15:59:18.984 INFO:tasks.ceph:Generating config... 2024-04-24T15:59:18.985 INFO:tasks.ceph:[client] debug rbd = 20 2024-04-24T15:59:18.985 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-04-24T15:59:18.985 INFO:tasks.ceph:[client] log to stderr = False 2024-04-24T15:59:18.985 INFO:tasks.ceph:[client] rbd default features = 125 2024-04-24T15:59:18.985 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:18.985 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:18.985 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:18.985 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:18.986 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:18.986 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:18.986 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:18.986 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:18.986 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:18.986 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:18.986 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:18.986 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:18.986 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:18.986 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:18.987 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-24T15:59:18.987 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-24T15:59:18.987 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-24T15:59:18.987 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T15:59:18.987 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T15:59:18.987 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T15:59:18.987 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T15:59:18.987 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T15:59:18.987 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T15:59:18.987 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-04-24T15:59:18.987 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-24T15:59:18.987 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T15:59:18.987 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-24T15:59:18.987 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-24T15:59:18.987 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T15:59:18.988 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T15:59:18.988 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-24T15:59:18.988 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T15:59:18.988 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T15:59:18.988 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T15:59:18.988 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T15:59:18.988 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T15:59:18.988 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-04-24T15:59:18.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-04-24T15:59:19.064 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster2.keyring 2024-04-24T15:59:19.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-04-24T15:59:19.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-24T15:59:19.215 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.136')] 2024-04-24T15:59:19.216 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.136', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-04-24T15:59:19.216 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:19.217 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T15:59:19.273 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.136 --print /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool: generated fsid e6ccc4a4-622a-400f-a916-d56c405e9e43 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:setting min_mon_release = octopus 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:epoch 0 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:fsid e6ccc4a4-622a-400f-a916-d56c405e9e43 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:last_changed 2024-04-24T15:59:19.354996+0000 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:created 2024-04-24T15:59:19.354996+0000 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:min_mon_release 15 (octopus) 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:election_strategy: 1 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:0: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.a 2024-04-24T15:59:19.352 INFO:teuthology.orchestra.run.smithi136.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-04-24T15:59:19.353 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T15:59:19.410 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID e6ccc4a4-622a-400f-a916-d56c405e9e43... 2024-04-24T15:59:19.411 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-24T15:59:19.414 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-24T15:59:19.473 INFO:teuthology.orchestra.run.smithi042.stdout:[global] 2024-04-24T15:59:19.473 INFO:teuthology.orchestra.run.smithi042.stdout: chdir = "" 2024-04-24T15:59:19.473 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T15:59:19.473 INFO:teuthology.orchestra.run.smithi042.stdout: auth supported = cephx 2024-04-24T15:59:19.473 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.473 INFO:teuthology.orchestra.run.smithi042.stdout: filestore xattr use omap = true 2024-04-24T15:59:19.473 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.473 INFO:teuthology.orchestra.run.smithi042.stdout: mon clock drift allowed = 1.000 2024-04-24T15:59:19.474 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.474 INFO:teuthology.orchestra.run.smithi042.stdout: osd crush chooseleaf type = 0 2024-04-24T15:59:19.474 INFO:teuthology.orchestra.run.smithi042.stdout: auth debug = true 2024-04-24T15:59:19.474 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.474 INFO:teuthology.orchestra.run.smithi042.stdout: ms die on old message = true 2024-04-24T15:59:19.474 INFO:teuthology.orchestra.run.smithi042.stdout: ms die on bug = true 2024-04-24T15:59:19.474 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: mon pg warn max object skew = 0 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default size = 2 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd allow primary affinity = true 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd allow pg remap = true 2024-04-24T15:59:19.475 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on legacy crush tunables = false 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on crush straw calc version zero = false 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on no sortbitwise = false 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on osd down out interval zero = false 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on too few osds = false 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: mon_allow_pool_size_one = true 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: osd default data pool replay window = 5 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: mon allow pool delete = true 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.476 INFO:teuthology.orchestra.run.smithi042.stdout: mon cluster log file level = debug 2024-04-24T15:59:19.477 INFO:teuthology.orchestra.run.smithi042.stdout: debug asserts on shutdown = true 2024-04-24T15:59:19.477 INFO:teuthology.orchestra.run.smithi042.stdout: mon health detail to clog = false 2024-04-24T15:59:19.477 INFO:teuthology.orchestra.run.smithi042.stdout: mon host = 172.21.15.136 2024-04-24T15:59:19.477 INFO:teuthology.orchestra.run.smithi042.stdout: mon client directed command retry = 5 2024-04-24T15:59:19.477 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on pool no app = False 2024-04-24T15:59:19.477 INFO:teuthology.orchestra.run.smithi042.stdout: ms inject socket failures = 5000 2024-04-24T15:59:19.477 INFO:teuthology.orchestra.run.smithi042.stdout: fsid = e6ccc4a4-622a-400f-a916-d56c405e9e43 2024-04-24T15:59:19.477 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.477 INFO:teuthology.orchestra.run.smithi042.stdout:[osd] 2024-04-24T15:59:19.477 INFO:teuthology.orchestra.run.smithi042.stdout: osd journal size = 100 2024-04-24T15:59:19.478 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.478 INFO:teuthology.orchestra.run.smithi042.stdout: osd scrub load threshold = 5.0 2024-04-24T15:59:19.478 INFO:teuthology.orchestra.run.smithi042.stdout: osd scrub max interval = 600 2024-04-24T15:59:19.478 INFO:teuthology.orchestra.run.smithi042.stdout: osd mclock profile = high_recovery_ops 2024-04-24T15:59:19.478 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.478 INFO:teuthology.orchestra.run.smithi042.stdout: osd recover clone overlap = true 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: osd recovery max chunk = 1048576 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug shutdown = true 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug op order = true 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug verify stray on activate = true 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: osd open classes on start = true 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug pg log writeout = true 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: osd deep scrub update digest min age = 30 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: osd map max advance = 10 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: journal zero on create = true 2024-04-24T15:59:19.479 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.480 INFO:teuthology.orchestra.run.smithi042.stdout: filestore ondisk finisher threads = 3 2024-04-24T15:59:19.480 INFO:teuthology.orchestra.run.smithi042.stdout: filestore apply finisher threads = 3 2024-04-24T15:59:19.480 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.480 INFO:teuthology.orchestra.run.smithi042.stdout: bdev debug aio = true 2024-04-24T15:59:19.480 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug misdirected ops = true 2024-04-24T15:59:19.480 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore block size = 96636764160 2024-04-24T15:59:19.480 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore compression algorithm = zlib 2024-04-24T15:59:19.480 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore compression mode = aggressive 2024-04-24T15:59:19.480 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore fsck on mount = True 2024-04-24T15:59:19.480 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluefs = 1/20 2024-04-24T15:59:19.481 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluestore = 1/20 2024-04-24T15:59:19.481 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-04-24T15:59:19.481 INFO:teuthology.orchestra.run.smithi042.stdout: debug osd = 20 2024-04-24T15:59:19.481 INFO:teuthology.orchestra.run.smithi042.stdout: debug rocksdb = 4/10 2024-04-24T15:59:19.481 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T15:59:19.481 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd full ratio = 0.9 2024-04-24T15:59:19.481 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd nearfull ratio = 0.8 2024-04-24T15:59:19.482 INFO:teuthology.orchestra.run.smithi042.stdout: osd failsafe full ratio = 0.95 2024-04-24T15:59:19.482 INFO:teuthology.orchestra.run.smithi042.stdout: osd objectstore = bluestore 2024-04-24T15:59:19.482 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.482 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr] 2024-04-24T15:59:19.482 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-04-24T15:59:19.483 INFO:teuthology.orchestra.run.smithi042.stdout: debug mgr = 20 2024-04-24T15:59:19.483 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 20 2024-04-24T15:59:19.483 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 20 2024-04-24T15:59:19.483 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:59:19.483 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:59:19.483 INFO:teuthology.orchestra.run.smithi042.stdout: mgr/telemetry/nag = false 2024-04-24T15:59:19.484 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.484 INFO:teuthology.orchestra.run.smithi042.stdout:[mon] 2024-04-24T15:59:19.484 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-04-24T15:59:19.484 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 20 2024-04-24T15:59:19.484 INFO:teuthology.orchestra.run.smithi042.stdout: debug paxos = 20 2024-04-24T15:59:19.484 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 20 2024-04-24T15:59:19.485 INFO:teuthology.orchestra.run.smithi042.stdout: mon data avail warn = 5 2024-04-24T15:59:19.485 INFO:teuthology.orchestra.run.smithi042.stdout: mon mgr mkfs grace = 240 2024-04-24T15:59:19.485 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:59:19.485 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd reporter subtree level = osd 2024-04-24T15:59:19.485 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd prime pg temp = true 2024-04-24T15:59:19.485 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:59:19.486 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.486 INFO:teuthology.orchestra.run.smithi042.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T15:59:19.486 INFO:teuthology.orchestra.run.smithi042.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T15:59:19.486 INFO:teuthology.orchestra.run.smithi042.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T15:59:19.486 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.487 INFO:teuthology.orchestra.run.smithi042.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T15:59:19.487 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T15:59:19.487 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T15:59:19.487 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.487 INFO:teuthology.orchestra.run.smithi042.stdout: # 1m isn't quite enough 2024-04-24T15:59:19.487 INFO:teuthology.orchestra.run.smithi042.stdout: mon_down_mkfs_grace = 2m 2024-04-24T15:59:19.488 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.488 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_filestore_osds = false 2024-04-24T15:59:19.488 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T15:59:19.488 INFO:teuthology.orchestra.run.smithi042.stdout:[client] 2024-04-24T15:59:19.488 INFO:teuthology.orchestra.run.smithi042.stdout: rgw cache enabled = true 2024-04-24T15:59:19.488 INFO:teuthology.orchestra.run.smithi042.stdout: rgw enable ops log = true 2024-04-24T15:59:19.489 INFO:teuthology.orchestra.run.smithi042.stdout: rgw enable usage log = true 2024-04-24T15:59:19.489 INFO:teuthology.orchestra.run.smithi042.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T15:59:19.489 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T15:59:19.489 INFO:teuthology.orchestra.run.smithi042.stdout: debug rbd = 20 2024-04-24T15:59:19.489 INFO:teuthology.orchestra.run.smithi042.stdout: debug rbd_mirror = 15 2024-04-24T15:59:19.489 INFO:teuthology.orchestra.run.smithi042.stdout: log to stderr = False 2024-04-24T15:59:19.490 INFO:teuthology.orchestra.run.smithi042.stdout: rbd default features = 125 2024-04-24T15:59:19.490 INFO:teuthology.orchestra.run.smithi042.stdout:[mon.a] 2024-04-24T15:59:19.490 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.0] 2024-04-24T15:59:19.490 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.490 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.490 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.1] 2024-04-24T15:59:19.490 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.491 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.491 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.2] 2024-04-24T15:59:19.491 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.491 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.491 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.3] 2024-04-24T15:59:19.491 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.491 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.492 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.4] 2024-04-24T15:59:19.492 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.492 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.492 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.5] 2024-04-24T15:59:19.492 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.493 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.493 INFO:teuthology.orchestra.run.smithi042.stdout:[client.mirror.6] 2024-04-24T15:59:19.493 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.493 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.497 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-04-24T15:59:19.497 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2024-04-24T15:59:19.497 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T15:59:19.497 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2024-04-24T15:59:19.497 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.497 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2024-04-24T15:59:19.497 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.498 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2024-04-24T15:59:19.498 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.498 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2024-04-24T15:59:19.498 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2024-04-24T15:59:19.498 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.498 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2024-04-24T15:59:19.499 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2024-04-24T15:59:19.499 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.499 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T15:59:19.499 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2024-04-24T15:59:19.499 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.499 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T15:59:19.500 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T15:59:19.500 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.500 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2024-04-24T15:59:19.500 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.500 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2024-04-24T15:59:19.500 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2024-04-24T15:59:19.501 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2024-04-24T15:59:19.501 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2024-04-24T15:59:19.501 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2024-04-24T15:59:19.501 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2024-04-24T15:59:19.501 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2024-04-24T15:59:19.501 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T15:59:19.502 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T15:59:19.502 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2024-04-24T15:59:19.502 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.502 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T15:59:19.502 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.502 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2024-04-24T15:59:19.503 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.503 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2024-04-24T15:59:19.503 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.503 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2024-04-24T15:59:19.503 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2024-04-24T15:59:19.504 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2024-04-24T15:59:19.504 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = 172.21.15.136 2024-04-24T15:59:19.504 INFO:teuthology.orchestra.run.smithi136.stdout: mon client directed command retry = 5 2024-04-24T15:59:19.504 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on pool no app = False 2024-04-24T15:59:19.504 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject socket failures = 5000 2024-04-24T15:59:19.504 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = e6ccc4a4-622a-400f-a916-d56c405e9e43 2024-04-24T15:59:19.505 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.505 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2024-04-24T15:59:19.505 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2024-04-24T15:59:19.505 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.505 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2024-04-24T15:59:19.505 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2024-04-24T15:59:19.506 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2024-04-24T15:59:19.506 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.506 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2024-04-24T15:59:19.506 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2024-04-24T15:59:19.506 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.506 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2024-04-24T15:59:19.507 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2024-04-24T15:59:19.507 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2024-04-24T15:59:19.507 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.507 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2024-04-24T15:59:19.507 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2024-04-24T15:59:19.507 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.507 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2024-04-24T15:59:19.507 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.508 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2024-04-24T15:59:19.508 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.508 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2024-04-24T15:59:19.508 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.508 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2024-04-24T15:59:19.508 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2024-04-24T15:59:19.508 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.509 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2024-04-24T15:59:19.509 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2024-04-24T15:59:19.509 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore block size = 96636764160 2024-04-24T15:59:19.509 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore compression algorithm = zlib 2024-04-24T15:59:19.509 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore compression mode = aggressive 2024-04-24T15:59:19.509 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore fsck on mount = True 2024-04-24T15:59:19.510 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluefs = 1/20 2024-04-24T15:59:19.510 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluestore = 1/20 2024-04-24T15:59:19.510 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-24T15:59:19.510 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2024-04-24T15:59:19.510 INFO:teuthology.orchestra.run.smithi136.stdout: debug rocksdb = 4/10 2024-04-24T15:59:19.510 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T15:59:19.510 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd full ratio = 0.9 2024-04-24T15:59:19.511 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd nearfull ratio = 0.8 2024-04-24T15:59:19.511 INFO:teuthology.orchestra.run.smithi136.stdout: osd failsafe full ratio = 0.95 2024-04-24T15:59:19.511 INFO:teuthology.orchestra.run.smithi136.stdout: osd objectstore = bluestore 2024-04-24T15:59:19.511 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.511 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2024-04-24T15:59:19.511 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-24T15:59:19.511 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2024-04-24T15:59:19.511 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-04-24T15:59:19.512 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-04-24T15:59:19.512 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:59:19.512 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:59:19.512 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2024-04-24T15:59:19.512 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.512 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2024-04-24T15:59:19.512 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-04-24T15:59:19.512 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-04-24T15:59:19.513 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2024-04-24T15:59:19.513 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-04-24T15:59:19.513 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2024-04-24T15:59:19.513 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2024-04-24T15:59:19.513 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-04-24T15:59:19.513 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2024-04-24T15:59:19.513 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2024-04-24T15:59:19.513 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-04-24T15:59:19.514 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.514 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T15:59:19.514 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T15:59:19.514 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T15:59:19.514 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.514 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T15:59:19.514 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T15:59:19.515 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T15:59:19.515 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.515 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2024-04-24T15:59:19.515 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2024-04-24T15:59:19.515 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.515 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2024-04-24T15:59:19.515 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:19.516 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2024-04-24T15:59:19.516 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2024-04-24T15:59:19.516 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2024-04-24T15:59:19.516 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2024-04-24T15:59:19.516 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T15:59:19.516 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T15:59:19.517 INFO:teuthology.orchestra.run.smithi136.stdout: debug rbd = 20 2024-04-24T15:59:19.517 INFO:teuthology.orchestra.run.smithi136.stdout: debug rbd_mirror = 15 2024-04-24T15:59:19.517 INFO:teuthology.orchestra.run.smithi136.stdout: log to stderr = False 2024-04-24T15:59:19.517 INFO:teuthology.orchestra.run.smithi136.stdout: rbd default features = 125 2024-04-24T15:59:19.517 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2024-04-24T15:59:19.517 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.0] 2024-04-24T15:59:19.517 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.517 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.518 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.1] 2024-04-24T15:59:19.518 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.518 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.518 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.2] 2024-04-24T15:59:19.518 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.519 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.519 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.3] 2024-04-24T15:59:19.519 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.519 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.519 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.4] 2024-04-24T15:59:19.519 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.519 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.520 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.5] 2024-04-24T15:59:19.520 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.520 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.520 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.6] 2024-04-24T15:59:19.520 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-04-24T15:59:19.520 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-04-24T15:59:19.522 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-04-24T15:59:19.522 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-04-24T15:59:19.576 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T15:59:19.576 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:19.576 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-04-24T15:59:19.593 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:19.593 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-04-24T15:59:19.650 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi042.front.sepia.ceph.com 2024-04-24T15:59:19.650 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:59:19.650 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-24T15:59:19.650 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-24T15:59:19.699 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:59:19.699 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-24T15:59:19.761 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T15:59:19.761 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:19.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-24T15:59:19.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-24T15:59:19.799 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:19.799 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-24T15:59:19.856 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T15:59:19.856 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T15:59:19.857 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-24T15:59:19.920 INFO:teuthology.orchestra.run.smithi042.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-24T15:59:19.923 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T15:59:19.923 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T15:59:19.923 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-04-24T15:59:19.971 INFO:teuthology.orchestra.run.smithi042.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-04-24T15:59:19.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-04-24T15:59:20.027 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-04-24T15:59:20.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-24T15:59:20.121 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-24T15:59:20.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-24T15:59:20.214 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-04-24T15:59:20.227 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-24T15:59:20.307 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-04-24T15:59:20.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-24T15:59:20.401 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-04-24T15:59:20.414 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-24T15:59:20.493 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-04-24T15:59:20.506 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-24T15:59:20.587 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-04-24T15:59:20.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-24T15:59:20.679 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-04-24T15:59:20.692 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T15:59:20.692 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-24T15:59:20.692 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-04-24T15:59:20.759 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:59:20.759 INFO:tasks.ceph:role: cluster2.osd.0 2024-04-24T15:59:20.759 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T15:59:20.759 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T15:59:20.833 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:59:20.834 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:59:20.834 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:59:20.834 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:59:20.834 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:59:20.834 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:59:20.834 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:59:20.834 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:59:20.834 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:59:20.834 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:59:21.313 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-04-24T15:59:21.315 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-04-24T15:59:21.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-04-24T15:59:21.431 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-04-24T15:59:21.464 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-04-24T15:59:21.530 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:59:21.530 INFO:tasks.ceph:role: cluster2.osd.1 2024-04-24T15:59:21.530 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T15:59:21.530 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T15:59:21.605 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:59:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:59:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:59:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:59:21.605 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:59:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:59:21.605 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:59:21.605 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:59:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:59:21.605 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:59:22.034 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-04-24T15:59:22.036 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-04-24T15:59:22.036 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-04-24T15:59:22.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-04-24T15:59:22.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-04-24T15:59:22.271 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-24T15:59:22.272 INFO:tasks.ceph:role: cluster2.osd.2 2024-04-24T15:59:22.272 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T15:59:22.272 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T15:59:22.348 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T15:59:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T15:59:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T15:59:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-24T15:59:22.348 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T15:59:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-04-24T15:59:22.348 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T15:59:22.348 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-24T15:59:22.348 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T15:59:22.348 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T15:59:22.826 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-04-24T15:59:22.827 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-04-24T15:59:22.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-04-24T15:59:22.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-04-24T15:59:22.998 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T15:59:23.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:23.084+0000 7f5b2308e540 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-04-24T15:59:23.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:23.085+0000 7f5b2308e540 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-04-24T15:59:23.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:23.085+0000 7f5b2308e540 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-24T15:59:23.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:23.085+0000 7f5b2308e540 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-24T15:59:23.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:23.085+0000 7f5b2308e540 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-24T15:59:23.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:23.085+0000 7f5b2308e540 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-04-24T15:59:23.610 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:23.612+0000 7f5b2308e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:59:24.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:24.383+0000 7f5b2308e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:59:25.141 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:25.143+0000 7f5b2308e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:59:25.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-04-24T15:59:26.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T15:59:26.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:26.096+0000 7f81001e6540 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-04-24T15:59:26.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:26.096+0000 7f81001e6540 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-04-24T15:59:26.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:26.096+0000 7f81001e6540 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-24T15:59:26.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:26.096+0000 7f81001e6540 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-24T15:59:26.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:26.096+0000 7f81001e6540 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-24T15:59:26.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:26.097+0000 7f81001e6540 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-04-24T15:59:26.621 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:26.623+0000 7f81001e6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:59:27.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:27.387+0000 7f81001e6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:59:28.148 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:28.150+0000 7f81001e6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:59:28.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-04-24T15:59:28.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T15:59:29.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:29.065+0000 7f8e3e029540 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-04-24T15:59:29.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:29.065+0000 7f8e3e029540 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-04-24T15:59:29.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:29.065+0000 7f8e3e029540 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-24T15:59:29.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:29.065+0000 7f8e3e029540 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-24T15:59:29.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:29.065+0000 7f8e3e029540 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-24T15:59:29.063 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:29.066+0000 7f8e3e029540 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-04-24T15:59:29.587 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:29.589+0000 7f8e3e029540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:59:30.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:30.357+0000 7f8e3e029540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:59:31.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-24T15:59:31.119+0000 7f8e3e029540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-04-24T15:59:31.916 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-04-24T15:59:31.948 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T15:59:31.948 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:59:31.948 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-04-24T15:59:31.986 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:31.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-04-24T15:59:32.019 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:32.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-04-24T15:59:32.089 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:32.089 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-04-24T15:59:32.158 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-04-24T15:59:32.161 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-04-24T15:59:32.184 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:32.184 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-04-24T15:59:32.217 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:32.217 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-04-24T15:59:32.274 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:32.274 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-04-24T15:59:32.332 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:32.332 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-04-24T15:59:32.390 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:32.390 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-04-24T15:59:32.447 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:32.447 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-04-24T15:59:32.504 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:32.504 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-04-24T15:59:32.561 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:32.561 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-04-24T15:59:32.617 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T15:59:32.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo tee -a /etc/ceph/cluster2.keyring 2024-04-24T15:59:32.683 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.x] 2024-04-24T15:59:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBXLClmYQwRNxAAQ9p/NYBXguOZXfUvCs2zrg== 2024-04-24T15:59:32.683 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBbLClm2vIaBRAAgVMmgHQu0AKSLX/EyuGtfA== 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBeLClm1pTPBRAAEj2leSQ2yYJQHVCU6Eu8sQ== 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBhLClmYfbuAxAAjxjpW3yfO/rAFvlOXCmo8Q== 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBXLClmQCEaOhAAo2CqN7GGnHt1kIliN0/wSA== 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror] 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBYLClmjenMARAAobVpxG+L7u0Ne8FQ08406A== 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.0] 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBYLClmMLdkBxAAqwr3cOQBSAGUV4ha8SweNg== 2024-04-24T15:59:32.684 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.1] 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBYLClmDrPzDBAA9hVmw8sMbJqCsk68XEVy0w== 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.2] 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBYLClmU5CDEhAAIV54qW2bH5HvNeEyLUlVsA== 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.3] 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBYLClmbYMaGBAAbRKOUdsdcYTmVMe78SEO1w== 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.4] 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBYLClmvT6hHRAAcEEhDLPZDNh/NUZQWP5XbA== 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.5] 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBYLClmU/ovIxAA8ChQd84I2oBPFdDJBGBQTw== 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout:[client.mirror.6] 2024-04-24T15:59:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBYLClmoOyyKBAAOg3Z38XEg1VnEcjxrs/XtA== 2024-04-24T15:59:32.686 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T15:59:32.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:59:32.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:59:32.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T15:59:32.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:59:32.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:59:33.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:59:33.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:59:33.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:59:33.192 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:59:33.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:59:33.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:59:33.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T15:59:33.403 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T15:59:33.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-04-24T15:59:33.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-04-24T15:59:33.523 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-04-24T15:59:33.550 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-04-24T15:59:33.607 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-04-24T15:59:33.608 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-04-24T15:59:33.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-04-24T15:59:33.650 INFO:tasks.ceph.cluster2.mon.a:Started 2024-04-24T15:59:33.650 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-04-24T15:59:33.650 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-04-24T15:59:33.650 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-04-24T15:59:33.653 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-04-24T15:59:33.653 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T15:59:33.653 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster cluster2 osd crush tunables default 2024-04-24T15:59:33.862 INFO:teuthology.orchestra.run.smithi136.stderr:adjusted tunables profile to default 2024-04-24T15:59:33.873 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-04-24T15:59:33.873 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:33.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-04-24T15:59:33.900 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:33.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-04-24T15:59:33.967 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T15:59:33.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-04-24T15:59:34.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster cluster2 osd new cfb39152-2b32-4f85-9b4a-f2ffcc9fed42 0 2024-04-24T15:59:34.113 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:34.114+0000 7fc8b4d61180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T15:59:34.202 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:34.201+0000 7fc8b4d61180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T15:59:34.279 INFO:teuthology.orchestra.run.smithi136.stdout:0 2024-04-24T15:59:34.279 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:34.281+0000 7fc8b4d61180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T15:59:34.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster cluster2 osd new 80d9ea77-6b87-44c8-bf04-a95b22ad41f5 1 2024-04-24T15:59:34.353 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:34.354+0000 7fc8b4d61180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T15:59:34.496 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-04-24T15:59:34.508 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster cluster2 osd new 240de483-dd6c-4282-ad4c-f5fdb8a32545 2 2024-04-24T15:59:34.525 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:34.527+0000 7fc8b4d61180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T15:59:34.679 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:34.680+0000 7fc8b4d61180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T15:59:34.716 INFO:teuthology.orchestra.run.smithi136.stdout:2 2024-04-24T15:59:34.728 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-04-24T15:59:34.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-04-24T15:59:34.730 INFO:tasks.ceph.cluster2.osd.0:Started 2024-04-24T15:59:34.730 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-04-24T15:59:34.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-04-24T15:59:34.732 INFO:tasks.ceph.cluster2.osd.1:Started 2024-04-24T15:59:34.732 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-04-24T15:59:34.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-04-24T15:59:34.734 INFO:tasks.ceph.cluster2.osd.2:Started 2024-04-24T15:59:34.735 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T15:59:34.754 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:34.755+0000 7fc8b4d61180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T15:59:34.890 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:34.891+0000 7fc8b4d61180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T15:59:34.942 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:34.943 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":5,"fsid":"e6ccc4a4-622a-400f-a916-d56c405e9e43","created":"2024-04-24T15:59:33.767454+0000","modified":"2024-04-24T15:59:34.715818+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T15:59:34.715818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"cfb39152-2b32-4f85-9b4a-f2ffcc9fed42","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"80d9ea77-6b87-44c8-bf04-a95b22ad41f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"240de483-dd6c-4282-ad4c-f5fdb8a32545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:59:34.954 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-04-24T15:59:34.955 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T15:59:34.971 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:34.973+0000 7fc8b4d61180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T15:59:35.144 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:35.146+0000 7fc8b4d61180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T15:59:35.257 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-24T15:59:35.283 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:35.285+0000 7fc8b4d61180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T15:59:35.339 INFO:tasks.ceph.cluster2.osd.0.smithi136.stderr:2024-04-24T15:59:35.341+0000 7f4efc581540 -1 Falling back to public interface 2024-04-24T15:59:35.340 INFO:tasks.ceph.cluster2.osd.2.smithi136.stderr:2024-04-24T15:59:35.341+0000 7f564ce0d540 -1 Falling back to public interface 2024-04-24T15:59:35.340 INFO:tasks.ceph.cluster2.osd.2.smithi136.stderr:2024-04-24T15:59:35.341+0000 7f564722f700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-04-24T15:59:35.345 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T15:59:35.347+0000 7f7d47f9a540 -1 Falling back to public interface 2024-04-24T15:59:35.441 INFO:teuthology.misc.health.smithi136.stdout: 2024-04-24T15:59:35.441 INFO:teuthology.misc.health.smithi136.stdout:{"epoch":5,"fsid":"e6ccc4a4-622a-400f-a916-d56c405e9e43","created":"2024-04-24T15:59:33.767454+0000","modified":"2024-04-24T15:59:34.715818+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T15:59:34.715818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"cfb39152-2b32-4f85-9b4a-f2ffcc9fed42","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"80d9ea77-6b87-44c8-bf04-a95b22ad41f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"240de483-dd6c-4282-ad4c-f5fdb8a32545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:59:35.451 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-24T15:59:35.640 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:35.642+0000 7fc8b4d61180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T15:59:35.876 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:35.878+0000 7fc8b4d61180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T15:59:36.132 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:36.134+0000 7fc8b4d61180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T15:59:36.211 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:36.212+0000 7fc8b4d61180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T15:59:36.900 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:36.901+0000 7fc8b4d61180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T15:59:36.984 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:36.985+0000 7fc8b4d61180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T15:59:37.122 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:37.124+0000 7fc8b4d61180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T15:59:37.702 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:37.703+0000 7fc8b4d61180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T15:59:37.866 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:37.868+0000 7fc8b4d61180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T15:59:38.033 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:38.034+0000 7fc8b4d61180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T15:59:38.777 INFO:tasks.ceph.cluster2.osd.2.smithi136.stderr:2024-04-24T15:59:38.779+0000 7f564ce0d540 -1 osd.2 0 log_to_monitors true 2024-04-24T15:59:38.801 INFO:tasks.ceph.cluster2.osd.0.smithi136.stderr:2024-04-24T15:59:38.803+0000 7f4efc581540 -1 osd.0 0 log_to_monitors true 2024-04-24T15:59:38.829 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:38.831+0000 7fc8b4d61180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T15:59:38.838 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T15:59:38.841+0000 7f7d47f9a540 -1 osd.1 0 log_to_monitors true 2024-04-24T15:59:39.067 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:39.069+0000 7fc8b4d61180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T15:59:39.148 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:39.149+0000 7fc8b4d61180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T15:59:39.318 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:39.319+0000 7fc8b4d61180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T15:59:40.338 INFO:tasks.ceph.cluster2.mgr.x.smithi042.stderr:2024-04-24T15:59:40.339+0000 7fc873897700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-24T15:59:41.754 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-24T15:59:42.090 INFO:teuthology.misc.health.smithi136.stdout: 2024-04-24T15:59:42.091 INFO:teuthology.misc.health.smithi136.stdout:{"epoch":8,"fsid":"e6ccc4a4-622a-400f-a916-d56c405e9e43","created":"2024-04-24T15:59:33.767454+0000","modified":"2024-04-24T15:59:40.834815+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T15:59:34.715818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"cfb39152-2b32-4f85-9b4a-f2ffcc9fed42","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"80d9ea77-6b87-44c8-bf04-a95b22ad41f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"240de483-dd6c-4282-ad4c-f5fdb8a32545","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:59:42.104 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-24T15:59:42.743 INFO:tasks.ceph.cluster2.osd.0.smithi136.stderr:2024-04-24T15:59:42.745+0000 7f4ef319c700 -1 osd.0 0 waiting for initial osdmap 2024-04-24T15:59:42.747 INFO:tasks.ceph.cluster2.osd.0.smithi136.stderr:2024-04-24T15:59:42.750+0000 7f4eebf93700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:59:42.772 INFO:tasks.ceph.cluster2.osd.2.smithi136.stderr:2024-04-24T15:59:42.774+0000 7f564522b700 -1 osd.2 0 waiting for initial osdmap 2024-04-24T15:59:42.774 INFO:tasks.ceph.cluster2.osd.2.smithi136.stderr:2024-04-24T15:59:42.777+0000 7f563c81f700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:59:43.729 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T15:59:43.732+0000 7f7d3ebb5700 -1 osd.1 0 waiting for initial osdmap 2024-04-24T15:59:43.732 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T15:59:43.735+0000 7f7d379ac700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T15:59:48.407 DEBUG:teuthology.orchestra.run.smithi136:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-24T15:59:48.719 INFO:teuthology.misc.health.smithi136.stdout: 2024-04-24T15:59:48.720 INFO:teuthology.misc.health.smithi136.stdout:{"epoch":13,"fsid":"e6ccc4a4-622a-400f-a916-d56c405e9e43","created":"2024-04-24T15:59:33.767454+0000","modified":"2024-04-24T15:59:47.755872+0000","last_up_change":"2024-04-24T15:59:44.747907+0000","last_in_change":"2024-04-24T15:59:34.715818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T15:59:45.382463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cfb39152-2b32-4f85-9b4a-f2ffcc9fed42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":40663},{"type":"v1","addr":"172.21.15.136:6802","nonce":40663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":40663},{"type":"v1","addr":"172.21.15.136:6806","nonce":40663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":40663},{"type":"v1","addr":"172.21.15.136:6814","nonce":40663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":40663},{"type":"v1","addr":"172.21.15.136:6810","nonce":40663}]},"public_addr":"172.21.15.136:6802/40663","cluster_addr":"172.21.15.136:6806/40663","heartbeat_back_addr":"172.21.15.136:6814/40663","heartbeat_front_addr":"172.21.15.136:6810/40663","state":["exists","up"]},{"osd":1,"uuid":"80d9ea77-6b87-44c8-bf04-a95b22ad41f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":40665},{"type":"v1","addr":"172.21.15.136:6817","nonce":40665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":40665},{"type":"v1","addr":"172.21.15.136:6819","nonce":40665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":40665},{"type":"v1","addr":"172.21.15.136:6823","nonce":40665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":40665},{"type":"v1","addr":"172.21.15.136:6821","nonce":40665}]},"public_addr":"172.21.15.136:6817/40665","cluster_addr":"172.21.15.136:6819/40665","heartbeat_back_addr":"172.21.15.136:6823/40665","heartbeat_front_addr":"172.21.15.136:6821/40665","state":["exists","up"]},{"osd":2,"uuid":"240de483-dd6c-4282-ad4c-f5fdb8a32545","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":40664},{"type":"v1","addr":"172.21.15.136:6803","nonce":40664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":40664},{"type":"v1","addr":"172.21.15.136:6807","nonce":40664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":40664},{"type":"v1","addr":"172.21.15.136:6815","nonce":40664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":40664},{"type":"v1","addr":"172.21.15.136:6811","nonce":40664}]},"public_addr":"172.21.15.136:6803/40664","cluster_addr":"172.21.15.136:6807/40664","heartbeat_back_addr":"172.21.15.136:6815/40664","heartbeat_front_addr":"172.21.15.136:6811/40664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.824415+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.805861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.799092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:59:48.732 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-24T15:59:48.733 INFO:tasks.ceph:Creating RBD pool 2024-04-24T15:59:48.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-04-24T15:59:49.772 INFO:teuthology.orchestra.run.smithi136.stderr:pool 'rbd' created 2024-04-24T15:59:49.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-24T15:59:50.772 INFO:teuthology.orchestra.run.smithi136.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-24T15:59:50.786 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-04-24T15:59:50.787 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-04-24T15:59:50.787 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-04-24T15:59:50.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-04-24T15:59:50.787 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T15:59:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:51.169 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6826","nonce":44006},{"type":"v1","addr":"172.21.15.42:6827","nonce":44006}]},"active_addr":"172.21.15.42:6827/44006","active_change":"2024-04-24T15:59:39.334706+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":849951461}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":3459753264}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":1108028882}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":3772578913}]}]} 2024-04-24T15:59:51.173 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-04-24T15:59:51.173 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-04-24T15:59:51.174 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T15:59:51.483 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:51.483 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":15,"fsid":"e6ccc4a4-622a-400f-a916-d56c405e9e43","created":"2024-04-24T15:59:33.767454+0000","modified":"2024-04-24T15:59:50.765044+0000","last_up_change":"2024-04-24T15:59:44.747907+0000","last_in_change":"2024-04-24T15:59:34.715818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T15:59:45.382463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T15:59:49.063456+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cfb39152-2b32-4f85-9b4a-f2ffcc9fed42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":40663},{"type":"v1","addr":"172.21.15.136:6802","nonce":40663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":40663},{"type":"v1","addr":"172.21.15.136:6806","nonce":40663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":40663},{"type":"v1","addr":"172.21.15.136:6814","nonce":40663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":40663},{"type":"v1","addr":"172.21.15.136:6810","nonce":40663}]},"public_addr":"172.21.15.136:6802/40663","cluster_addr":"172.21.15.136:6806/40663","heartbeat_back_addr":"172.21.15.136:6814/40663","heartbeat_front_addr":"172.21.15.136:6810/40663","state":["exists","up"]},{"osd":1,"uuid":"80d9ea77-6b87-44c8-bf04-a95b22ad41f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":40665},{"type":"v1","addr":"172.21.15.136:6817","nonce":40665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":40665},{"type":"v1","addr":"172.21.15.136:6819","nonce":40665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":40665},{"type":"v1","addr":"172.21.15.136:6823","nonce":40665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":40665},{"type":"v1","addr":"172.21.15.136:6821","nonce":40665}]},"public_addr":"172.21.15.136:6817/40665","cluster_addr":"172.21.15.136:6819/40665","heartbeat_back_addr":"172.21.15.136:6823/40665","heartbeat_front_addr":"172.21.15.136:6821/40665","state":["exists","up"]},{"osd":2,"uuid":"240de483-dd6c-4282-ad4c-f5fdb8a32545","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":40664},{"type":"v1","addr":"172.21.15.136:6803","nonce":40664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":40664},{"type":"v1","addr":"172.21.15.136:6807","nonce":40664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":40664},{"type":"v1","addr":"172.21.15.136:6815","nonce":40664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":40664},{"type":"v1","addr":"172.21.15.136:6811","nonce":40664}]},"public_addr":"172.21.15.136:6803/40664","cluster_addr":"172.21.15.136:6807/40664","heartbeat_back_addr":"172.21.15.136:6815/40664","heartbeat_front_addr":"172.21.15.136:6811/40664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.824415+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.805861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.799092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:59:51.497 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-04-24T15:59:51.498 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T15:59:51.811 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:51.812 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":16,"fsid":"e6ccc4a4-622a-400f-a916-d56c405e9e43","created":"2024-04-24T15:59:33.767454+0000","modified":"2024-04-24T15:59:51.768246+0000","last_up_change":"2024-04-24T15:59:44.747907+0000","last_in_change":"2024-04-24T15:59:34.715818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T15:59:45.382463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T15:59:49.063456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cfb39152-2b32-4f85-9b4a-f2ffcc9fed42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":40663},{"type":"v1","addr":"172.21.15.136:6802","nonce":40663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":40663},{"type":"v1","addr":"172.21.15.136:6806","nonce":40663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":40663},{"type":"v1","addr":"172.21.15.136:6814","nonce":40663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":40663},{"type":"v1","addr":"172.21.15.136:6810","nonce":40663}]},"public_addr":"172.21.15.136:6802/40663","cluster_addr":"172.21.15.136:6806/40663","heartbeat_back_addr":"172.21.15.136:6814/40663","heartbeat_front_addr":"172.21.15.136:6810/40663","state":["exists","up"]},{"osd":1,"uuid":"80d9ea77-6b87-44c8-bf04-a95b22ad41f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":40665},{"type":"v1","addr":"172.21.15.136:6817","nonce":40665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":40665},{"type":"v1","addr":"172.21.15.136:6819","nonce":40665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":40665},{"type":"v1","addr":"172.21.15.136:6823","nonce":40665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":40665},{"type":"v1","addr":"172.21.15.136:6821","nonce":40665}]},"public_addr":"172.21.15.136:6817/40665","cluster_addr":"172.21.15.136:6819/40665","heartbeat_back_addr":"172.21.15.136:6823/40665","heartbeat_front_addr":"172.21.15.136:6821/40665","state":["exists","up"]},{"osd":2,"uuid":"240de483-dd6c-4282-ad4c-f5fdb8a32545","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":40664},{"type":"v1","addr":"172.21.15.136:6803","nonce":40664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":40664},{"type":"v1","addr":"172.21.15.136:6807","nonce":40664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":40664},{"type":"v1","addr":"172.21.15.136:6815","nonce":40664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":40664},{"type":"v1","addr":"172.21.15.136:6811","nonce":40664}]},"public_addr":"172.21.15.136:6803/40664","cluster_addr":"172.21.15.136:6807/40664","heartbeat_back_addr":"172.21.15.136:6815/40664","heartbeat_front_addr":"172.21.15.136:6811/40664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.824415+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.805861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.799092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T15:59:51.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-04-24T15:59:51.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-04-24T15:59:51.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-04-24T15:59:51.987 INFO:teuthology.orchestra.run.smithi136.stdout:42949672967 2024-04-24T15:59:51.988 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-24T15:59:51.989 INFO:teuthology.orchestra.run.smithi136.stdout:38654705670 2024-04-24T15:59:51.989 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-24T15:59:51.990 INFO:teuthology.orchestra.run.smithi136.stdout:38654705670 2024-04-24T15:59:51.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-24T15:59:52.335 INFO:teuthology.orchestra.run.smithi136.stdout:42949672966 2024-04-24T15:59:52.354 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672967 got 42949672966 for osd.1 2024-04-24T15:59:52.383 INFO:teuthology.orchestra.run.smithi136.stdout:38654705669 2024-04-24T15:59:52.387 INFO:teuthology.orchestra.run.smithi136.stdout:38654705669 2024-04-24T15:59:52.399 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705669 for osd.0 2024-04-24T15:59:52.402 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705669 for osd.2 2024-04-24T15:59:53.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-24T15:59:53.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-24T15:59:53.403 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-24T15:59:53.718 INFO:teuthology.orchestra.run.smithi136.stdout:42949672966 2024-04-24T15:59:53.732 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672967 got 42949672966 for osd.1 2024-04-24T15:59:53.755 INFO:teuthology.orchestra.run.smithi136.stdout:38654705669 2024-04-24T15:59:53.769 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705669 for osd.0 2024-04-24T15:59:53.784 INFO:teuthology.orchestra.run.smithi136.stdout:38654705670 2024-04-24T15:59:53.799 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705670 for osd.2 2024-04-24T15:59:53.799 DEBUG:teuthology.parallel:result is None 2024-04-24T15:59:54.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-24T15:59:54.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-24T15:59:55.074 INFO:teuthology.orchestra.run.smithi136.stdout:42949672967 2024-04-24T15:59:55.088 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672967 got 42949672967 for osd.1 2024-04-24T15:59:55.088 DEBUG:teuthology.parallel:result is None 2024-04-24T15:59:55.105 INFO:teuthology.orchestra.run.smithi136.stdout:38654705670 2024-04-24T15:59:55.119 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705670 got 38654705670 for osd.0 2024-04-24T15:59:55.119 DEBUG:teuthology.parallel:result is None 2024-04-24T15:59:55.119 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-24T15:59:55.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T15:59:55.470 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:55.472 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-24T15:59:55.483 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-24T15:59:55.341520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":892588,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282222932,"statfs":{"total":289910292480,"available":288996282368,"internally_reserved":0,"allocated":532480,"data_stored":1297292,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.572444"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.777319+0000","last_change":"2024-04-24T15:59:50.776430+0000","last_active":"2024-04-24T15:59:50.777319+0000","last_peered":"2024-04-24T15:59:50.777319+0000","last_clean":"2024-04-24T15:59:50.777319+0000","last_became_active":"2024-04-24T15:59:50.776148+0000","last_became_peered":"2024-04-24T15:59:50.776148+0000","last_unstale":"2024-04-24T15:59:50.777319+0000","last_undegraded":"2024-04-24T15:59:50.777319+0000","last_fullsized":"2024-04-24T15:59:50.777319+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:30:20.365824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.776857+0000","last_change":"2024-04-24T15:59:50.774597+0000","last_active":"2024-04-24T15:59:50.776857+0000","last_peered":"2024-04-24T15:59:50.776857+0000","last_clean":"2024-04-24T15:59:50.776857+0000","last_became_active":"2024-04-24T15:59:50.774160+0000","last_became_peered":"2024-04-24T15:59:50.774160+0000","last_unstale":"2024-04-24T15:59:50.776857+0000","last_undegraded":"2024-04-24T15:59:50.776857+0000","last_fullsized":"2024-04-24T15:59:50.776857+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:12.412466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.777979+0000","last_change":"2024-04-24T15:59:50.777598+0000","last_active":"2024-04-24T15:59:50.777979+0000","last_peered":"2024-04-24T15:59:50.777979+0000","last_clean":"2024-04-24T15:59:50.777979+0000","last_became_active":"2024-04-24T15:59:50.777322+0000","last_became_peered":"2024-04-24T15:59:50.777322+0000","last_unstale":"2024-04-24T15:59:50.777979+0000","last_undegraded":"2024-04-24T15:59:50.777979+0000","last_fullsized":"2024-04-24T15:59:50.777979+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:54:24.332122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.776617+0000","last_change":"2024-04-24T15:59:50.775501+0000","last_active":"2024-04-24T15:59:50.776617+0000","last_peered":"2024-04-24T15:59:50.776617+0000","last_clean":"2024-04-24T15:59:50.776617+0000","last_became_active":"2024-04-24T15:59:50.775157+0000","last_became_peered":"2024-04-24T15:59:50.775157+0000","last_unstale":"2024-04-24T15:59:50.776617+0000","last_undegraded":"2024-04-24T15:59:50.776617+0000","last_fullsized":"2024-04-24T15:59:50.776617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:43:00.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":139,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.772336+0000","last_change":"2024-04-24T15:59:46.916688+0000","last_active":"2024-04-24T15:59:50.772336+0000","last_peered":"2024-04-24T15:59:50.772336+0000","last_clean":"2024-04-24T15:59:50.772336+0000","last_became_active":"2024-04-24T15:59:46.915637+0000","last_became_peered":"2024-04-24T15:59:46.915637+0000","last_unstale":"2024-04-24T15:59:50.772336+0000","last_undegraded":"2024-04-24T15:59:50.772336+0000","last_fullsized":"2024-04-24T15:59:50.772336+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_clean_scrub_stamp":"2024-04-24T15:59:45.750019+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:30:29.745322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.777127+0000","last_change":"2024-04-24T15:59:50.775216+0000","last_active":"2024-04-24T15:59:50.777127+0000","last_peered":"2024-04-24T15:59:50.777127+0000","last_clean":"2024-04-24T15:59:50.777127+0000","last_became_active":"2024-04-24T15:59:50.774935+0000","last_became_peered":"2024-04-24T15:59:50.774935+0000","last_unstale":"2024-04-24T15:59:50.777127+0000","last_undegraded":"2024-04-24T15:59:50.777127+0000","last_fullsized":"2024-04-24T15:59:50.777127+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:50.389462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.776642+0000","last_change":"2024-04-24T15:59:50.775383+0000","last_active":"2024-04-24T15:59:50.776642+0000","last_peered":"2024-04-24T15:59:50.776642+0000","last_clean":"2024-04-24T15:59:50.776642+0000","last_became_active":"2024-04-24T15:59:50.775119+0000","last_became_peered":"2024-04-24T15:59:50.775119+0000","last_unstale":"2024-04-24T15:59:50.776642+0000","last_undegraded":"2024-04-24T15:59:50.776642+0000","last_fullsized":"2024-04-24T15:59:50.776642+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:20:11.243695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.777093+0000","last_change":"2024-04-24T15:59:50.774230+0000","last_active":"2024-04-24T15:59:50.777093+0000","last_peered":"2024-04-24T15:59:50.777093+0000","last_clean":"2024-04-24T15:59:50.777093+0000","last_became_active":"2024-04-24T15:59:50.773938+0000","last_became_peered":"2024-04-24T15:59:50.773938+0000","last_unstale":"2024-04-24T15:59:50.777093+0000","last_undegraded":"2024-04-24T15:59:50.777093+0000","last_fullsized":"2024-04-24T15:59:50.777093+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:54.764334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.777525+0000","last_change":"2024-04-24T15:59:50.776460+0000","last_active":"2024-04-24T15:59:50.777525+0000","last_peered":"2024-04-24T15:59:50.777525+0000","last_clean":"2024-04-24T15:59:50.777525+0000","last_became_active":"2024-04-24T15:59:50.776173+0000","last_became_peered":"2024-04-24T15:59:50.776173+0000","last_unstale":"2024-04-24T15:59:50.777525+0000","last_undegraded":"2024-04-24T15:59:50.777525+0000","last_fullsized":"2024-04-24T15:59:50.777525+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:54:40.948714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672968,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":196608,"data_stored":629220,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":196608,"data_stored":629220,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":139264,"data_stored":38852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T15:59:55.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T15:59:55.798 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:55.801 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-24T15:59:55.812 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-04-24T15:59:55.341520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":892588,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282222932,"statfs":{"total":289910292480,"available":288996282368,"internally_reserved":0,"allocated":532480,"data_stored":1297292,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.572444"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.777319+0000","last_change":"2024-04-24T15:59:50.776430+0000","last_active":"2024-04-24T15:59:50.777319+0000","last_peered":"2024-04-24T15:59:50.777319+0000","last_clean":"2024-04-24T15:59:50.777319+0000","last_became_active":"2024-04-24T15:59:50.776148+0000","last_became_peered":"2024-04-24T15:59:50.776148+0000","last_unstale":"2024-04-24T15:59:50.777319+0000","last_undegraded":"2024-04-24T15:59:50.777319+0000","last_fullsized":"2024-04-24T15:59:50.777319+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:30:20.365824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.776857+0000","last_change":"2024-04-24T15:59:50.774597+0000","last_active":"2024-04-24T15:59:50.776857+0000","last_peered":"2024-04-24T15:59:50.776857+0000","last_clean":"2024-04-24T15:59:50.776857+0000","last_became_active":"2024-04-24T15:59:50.774160+0000","last_became_peered":"2024-04-24T15:59:50.774160+0000","last_unstale":"2024-04-24T15:59:50.776857+0000","last_undegraded":"2024-04-24T15:59:50.776857+0000","last_fullsized":"2024-04-24T15:59:50.776857+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:12.412466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.777979+0000","last_change":"2024-04-24T15:59:50.777598+0000","last_active":"2024-04-24T15:59:50.777979+0000","last_peered":"2024-04-24T15:59:50.777979+0000","last_clean":"2024-04-24T15:59:50.777979+0000","last_became_active":"2024-04-24T15:59:50.777322+0000","last_became_peered":"2024-04-24T15:59:50.777322+0000","last_unstale":"2024-04-24T15:59:50.777979+0000","last_undegraded":"2024-04-24T15:59:50.777979+0000","last_fullsized":"2024-04-24T15:59:50.777979+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:54:24.332122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.776617+0000","last_change":"2024-04-24T15:59:50.775501+0000","last_active":"2024-04-24T15:59:50.776617+0000","last_peered":"2024-04-24T15:59:50.776617+0000","last_clean":"2024-04-24T15:59:50.776617+0000","last_became_active":"2024-04-24T15:59:50.775157+0000","last_became_peered":"2024-04-24T15:59:50.775157+0000","last_unstale":"2024-04-24T15:59:50.776617+0000","last_undegraded":"2024-04-24T15:59:50.776617+0000","last_fullsized":"2024-04-24T15:59:50.776617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:43:00.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":139,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.772336+0000","last_change":"2024-04-24T15:59:46.916688+0000","last_active":"2024-04-24T15:59:50.772336+0000","last_peered":"2024-04-24T15:59:50.772336+0000","last_clean":"2024-04-24T15:59:50.772336+0000","last_became_active":"2024-04-24T15:59:46.915637+0000","last_became_peered":"2024-04-24T15:59:46.915637+0000","last_unstale":"2024-04-24T15:59:50.772336+0000","last_undegraded":"2024-04-24T15:59:50.772336+0000","last_fullsized":"2024-04-24T15:59:50.772336+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_clean_scrub_stamp":"2024-04-24T15:59:45.750019+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:30:29.745322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.777127+0000","last_change":"2024-04-24T15:59:50.775216+0000","last_active":"2024-04-24T15:59:50.777127+0000","last_peered":"2024-04-24T15:59:50.777127+0000","last_clean":"2024-04-24T15:59:50.777127+0000","last_became_active":"2024-04-24T15:59:50.774935+0000","last_became_peered":"2024-04-24T15:59:50.774935+0000","last_unstale":"2024-04-24T15:59:50.777127+0000","last_undegraded":"2024-04-24T15:59:50.777127+0000","last_fullsized":"2024-04-24T15:59:50.777127+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:50.389462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.776642+0000","last_change":"2024-04-24T15:59:50.775383+0000","last_active":"2024-04-24T15:59:50.776642+0000","last_peered":"2024-04-24T15:59:50.776642+0000","last_clean":"2024-04-24T15:59:50.776642+0000","last_became_active":"2024-04-24T15:59:50.775119+0000","last_became_peered":"2024-04-24T15:59:50.775119+0000","last_unstale":"2024-04-24T15:59:50.776642+0000","last_undegraded":"2024-04-24T15:59:50.776642+0000","last_fullsized":"2024-04-24T15:59:50.776642+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:20:11.243695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.777093+0000","last_change":"2024-04-24T15:59:50.774230+0000","last_active":"2024-04-24T15:59:50.777093+0000","last_peered":"2024-04-24T15:59:50.777093+0000","last_clean":"2024-04-24T15:59:50.777093+0000","last_became_active":"2024-04-24T15:59:50.773938+0000","last_became_peered":"2024-04-24T15:59:50.773938+0000","last_unstale":"2024-04-24T15:59:50.777093+0000","last_undegraded":"2024-04-24T15:59:50.777093+0000","last_fullsized":"2024-04-24T15:59:50.777093+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:54.764334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-24T15:59:50.777525+0000","last_change":"2024-04-24T15:59:50.776460+0000","last_active":"2024-04-24T15:59:50.777525+0000","last_peered":"2024-04-24T15:59:50.777525+0000","last_clean":"2024-04-24T15:59:50.777525+0000","last_became_active":"2024-04-24T15:59:50.776173+0000","last_became_peered":"2024-04-24T15:59:50.776173+0000","last_unstale":"2024-04-24T15:59:50.777525+0000","last_undegraded":"2024-04-24T15:59:50.777525+0000","last_fullsized":"2024-04-24T15:59:50.777525+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:54:40.948714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672968,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":196608,"data_stored":629220,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":196608,"data_stored":629220,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":139264,"data_stored":38852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T15:59:55.813 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-24T15:59:55.813 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-04-24T15:59:55.813 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-04-24T15:59:55.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-04-24T15:59:56.167 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T15:59:56.167 INFO:teuthology.orchestra.run.smithi136.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T15:59:56.180 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-04-24T15:59:56.181 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T15:59:56.191 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T15:59:56.191 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T15:59:56.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T15:59:56.595 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.4 2024-04-24T15:59:56.607 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T15:59:57.013 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.5 2024-04-24T15:59:57.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T15:59:57.428 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.6 2024-04-24T15:59:57.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T15:59:57.842 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.4 2024-04-24T15:59:57.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T15:59:58.261 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.5 2024-04-24T15:59:58.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T15:59:58.675 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.6 2024-04-24T15:59:58.687 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T15:59:58.695 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T15:59:58.695 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T15:59:58.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T15:59:59.102 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror 2024-04-24T15:59:59.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T15:59:59.519 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.0 2024-04-24T15:59:59.531 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T15:59:59.939 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.1 2024-04-24T15:59:59.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T16:00:00.354 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.2 2024-04-24T16:00:00.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T16:00:00.770 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.3 2024-04-24T16:00:00.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T16:00:01.185 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror 2024-04-24T16:00:01.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T16:00:01.602 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.0 2024-04-24T16:00:01.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T16:00:02.017 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.1 2024-04-24T16:00:02.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T16:00:02.436 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.2 2024-04-24T16:00:02.448 DEBUG:teuthology.orchestra.run.smithi136:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-04-24T16:00:02.854 INFO:teuthology.orchestra.run.smithi136.stderr:updated caps for client.mirror.3 2024-04-24T16:00:02.865 INFO:teuthology.run_tasks:Running task workunit... 2024-04-24T16:00:02.881 INFO:tasks.workunit:Pulling workunits from ref 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T16:00:02.881 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-24T16:00:02.882 DEBUG:teuthology.orchestra.run.smithi136:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T16:00:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T16:00:02.899 INFO:teuthology.orchestra.run.smithi136.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-04-24T16:00:02.900 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T16:00:02.956 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T16:00:02.956 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-04-24T16:00:03.013 INFO:tasks.workunit:timeout=3h 2024-04-24T16:00:03.013 INFO:tasks.workunit:cleanup=True 2024-04-24T16:00:03.014 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 330067dbce47655d1f762ef14d546d2fde16a6a9 2024-04-24T16:00:03.071 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-04-24T16:00:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:Note: switching to '330067dbce47655d1f762ef14d546d2fde16a6a9'. 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:state without impacting any branches by switching back to a branch. 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: git switch -c 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:Or undo this operation with: 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2024-04-24T16:00:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: git switch - 2024-04-24T16:00:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2024-04-24T16:00:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-24T16:00:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2024-04-24T16:00:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:HEAD is now at 330067dbce4 Merge PR #54172 into wip-yuriw-testing-20240423.151325-quincy 2024-04-24T16:00:53.540 DEBUG:teuthology.orchestra.run.smithi136:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-04-24T16:00:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-24T16:00:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-24T16:00:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-24T16:00:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-24T16:00:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-24T16:00:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-24T16:00:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-24T16:00:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-24T16:00:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-24T16:00:54.312 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-24T16:00:54.312 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-04-24T16:00:54.329 INFO:tasks.workunit:Running workunits matching rbd/compare_mirror_image_alternate_primary.sh on cluster1.client.mirror... 2024-04-24T16:00:54.330 INFO:tasks.workunit:Running workunit rbd/compare_mirror_image_alternate_primary.sh... 2024-04-24T16:00:54.330 DEBUG:teuthology.orchestra.run.smithi136:workunit test rbd/compare_mirror_image_alternate_primary.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=330067dbce47655d1f762ef14d546d2fde16a6a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_image_alternate_primary.sh 2024-04-24T16:00:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ IMAGE=image-alternate-primary 2024-04-24T16:00:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ MIRROR_IMAGE_MODE=snapshot 2024-04-24T16:00:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ MIRROR_POOL_MODE=image 2024-04-24T16:00:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ MOUNT=test-alternate-primary 2024-04-24T16:00:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-04-24T16:00:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ RBD_MIRROR_INSTANCES=1 2024-04-24T16:00:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ RBD_MIRROR_MODE=snapshot 2024-04-24T16:00:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ RBD_MIRROR_USE_EXISTING_CLUSTER=1 2024-04-24T16:00:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_image_alternate_primary.sh 2024-04-24T16:00:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-04-24T16:00:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ type xmlstarlet 2024-04-24T16:00:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ XMLSTARLET=xmlstarlet 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ RBD_MIRROR_INSTANCES=1 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CLUSTER1=cluster1 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CLUSTER2=cluster2 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ PEER_CLUSTER_SUFFIX= 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ POOL=mirror 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ PARENT_POOL=mirror_parent 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ NS1=ns1 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ NS2=ns2 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ TEMPDIR= 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ID=mirror 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ MIRROR_POOL_MODE=image 2024-04-24T16:00:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ MIRROR_IMAGE_MODE=snapshot 2024-04-24T16:00:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-04-24T16:00:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS='--id mirror' 2024-04-24T16:00:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ LAST_MIRROR_INSTANCE=0 2024-04-24T16:00:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_image_alternate_primary.sh 2024-04-24T16:00:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-04-24T16:00:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-04-24T16:00:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_BIN=. 2024-04-24T16:00:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_SRC=. 2024-04-24T16:00:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ '[' -e CMakeCache.txt ']' 2024-04-24T16:00:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ '[' 0 -gt 0 ']' 2024-04-24T16:00:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ setup 2024-04-24T16:00:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local c 2024-04-24T16:00:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-04-24T16:00:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ setup_tempdir 2024-04-24T16:00:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:00:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ mktemp -d 2024-04-24T16:00:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMPDIR=/tmp/tmp.gEigttWnJf 2024-04-24T16:00:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -z 1 ']' 2024-04-24T16:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ setup_pools cluster1 cluster2 2024-04-24T16:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local remote_cluster=cluster2 2024-04-24T16:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mon_map_file 2024-04-24T16:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mon_addr 2024-04-24T16:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local admin_key_file 2024-04-24T16:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local uuid 2024-04-24T16:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T16:00:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-04-24T16:00:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:pool 'mirror' already exists 2024-04-24T16:00:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T16:00:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-04-24T16:00:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:pool 'mirror_parent' already exists 2024-04-24T16:00:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T16:00:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 pool init mirror 2024-04-24T16:00:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T16:00:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-04-24T16:01:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T16:01:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-04-24T16:01:01.894 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-04-24T16:01:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-04-24T16:01:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-04-24T16:01:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-04-24T16:01:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-04-24T16:01:02.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -z ']' 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -z ']' 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ peer_add cluster1 mirror cluster2 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local client_cluster=cluster2 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local remote_cluster=cluster2 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local uuid_var_name 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local error_code 2024-04-24T16:01:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local peer_uuid 2024-04-24T16:01:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T16:01:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set +e 2024-04-24T16:01:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-04-24T16:01:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ peer_uuid=3baa6bca-9af1-461e-82f1-e6f3589cb5eb 2024-04-24T16:01:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ error_code=0 2024-04-24T16:01:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set -e 2024-04-24T16:01:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T16:01:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test 0 -eq 0 2024-04-24T16:01:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:01:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-04-24T16:01:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror_parent 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local client_cluster=cluster2 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local remote_cluster=cluster2 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local uuid_var_name 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local error_code 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local peer_uuid 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set +e 2024-04-24T16:01:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-04-24T16:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ peer_uuid=4e0e8d83-50b0-4456-9bde-588912e67181 2024-04-24T16:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ error_code=0 2024-04-24T16:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set -e 2024-04-24T16:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T16:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test 0 -eq 0 2024-04-24T16:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ setup_pools cluster2 cluster1 2024-04-24T16:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:01:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local remote_cluster=cluster1 2024-04-24T16:01:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mon_map_file 2024-04-24T16:01:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mon_addr 2024-04-24T16:01:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local admin_key_file 2024-04-24T16:01:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local uuid 2024-04-24T16:01:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T16:01:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-04-24T16:01:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:pool 'mirror' already exists 2024-04-24T16:01:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T16:01:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-04-24T16:01:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:pool 'mirror_parent' already exists 2024-04-24T16:01:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T16:01:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 pool init mirror 2024-04-24T16:01:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T16:01:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-04-24T16:01:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T16:01:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-04-24T16:01:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-04-24T16:01:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-04-24T16:01:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-04-24T16:01:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-04-24T16:01:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-04-24T16:01:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -z ']' 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -z ']' 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ peer_add cluster2 mirror cluster1 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local client_cluster=cluster1 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local remote_cluster=cluster1 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local uuid_var_name 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local error_code 2024-04-24T16:01:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local peer_uuid 2024-04-24T16:01:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T16:01:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set +e 2024-04-24T16:01:10.659 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-04-24T16:01:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ peer_uuid=e447c8fd-fbd6-4dd9-aa4a-6bf645fe09dc 2024-04-24T16:01:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ error_code=0 2024-04-24T16:01:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set -e 2024-04-24T16:01:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T16:01:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test 0 -eq 0 2024-04-24T16:01:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:01:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-04-24T16:01:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:01:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror_parent 2024-04-24T16:01:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local client_cluster=cluster1 2024-04-24T16:01:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local remote_cluster=cluster1 2024-04-24T16:01:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local uuid_var_name 2024-04-24T16:01:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local error_code 2024-04-24T16:01:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local peer_uuid 2024-04-24T16:01:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T16:01:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set +e 2024-04-24T16:01:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-04-24T16:01:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ peer_uuid=ef38ebbc-9f23-407d-9f6a-0ddf1155a4aa 2024-04-24T16:01:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ error_code=0 2024-04-24T16:01:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set -e 2024-04-24T16:01:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' 0 -eq 17 ']' 2024-04-24T16:01:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test 0 -eq 0 2024-04-24T16:01:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:01:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:01:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ start_mirrors cluster1 2024-04-24T16:01:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ seq 0 0 2024-04-24T16:01:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T16:01:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ start_mirror cluster1:0 2024-04-24T16:01:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1:0 2024-04-24T16:01:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local instance 2024-04-24T16:01:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-24T16:01:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local val=cluster1:0 2024-04-24T16:01:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster_var_name=cluster 2024-04-24T16:01:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local instance_var_name=instance 2024-04-24T16:01:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cluster=cluster1 2024-04-24T16:01:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ instance=0 2024-04-24T16:01:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-24T16:01:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ eval cluster=cluster1 2024-04-24T16:01:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ cluster=cluster1 2024-04-24T16:01:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ eval instance=0 2024-04-24T16:01:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ instance=0 2024-04-24T16:01:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test -n '' 2024-04-24T16:01:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-04-24T16:01:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ start_mirrors cluster2 2024-04-24T16:01:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:01:10.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ seq 0 0 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ start_mirror cluster2:0 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2:0 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local instance 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local val=cluster2:0 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster_var_name=cluster 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local instance_var_name=instance 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cluster=cluster2 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ instance=0 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' 0 = cluster2:0 ']' 2024-04-24T16:01:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ eval cluster=cluster2 2024-04-24T16:01:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ cluster=cluster2 2024-04-24T16:01:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ eval instance=0 2024-04-24T16:01:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ instance=0 2024-04-24T16:01:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test -n '' 2024-04-24T16:01:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ create_image_and_enable_mirror cluster1 mirror image-alternate-primary snapshot 10G 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mode=snapshot 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n snapshot ']' 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ create_image cluster1 mirror image-alternate-primary 10G 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local size=128 2024-04-24T16:01:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n 10G ']' 2024-04-24T16:01:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ size=10G 2024-04-24T16:01:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ shift 2024-04-24T16:01:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-alternate-primary 2024-04-24T16:01:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' image = image ']' 2024-04-24T16:01:10.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ enable_mirror cluster1 mirror image-alternate-primary snapshot 2024-04-24T16:01:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mode=snapshot 2024-04-24T16:01:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-alternate-primary snapshot 2024-04-24T16:01:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Mirroring enabled 2024-04-24T16:01:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 info mirror/image-alternate-primary 2024-04-24T16:01:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:rbd image 'image-alternate-primary': 2024-04-24T16:01:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: size 10 GiB in 2560 objects 2024-04-24T16:01:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: order 22 (4 MiB objects) 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: snapshot_count: 1 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: id: 10ab1bb1d227 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: block_name_prefix: rbd_data.10ab1bb1d227 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: format: 2 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: op_features: 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: flags: 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: create_timestamp: Wed Apr 24 16:01:10 2024 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: access_timestamp: Wed Apr 24 16:01:10 2024 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: modify_timestamp: Wed Apr 24 16:01:10 2024 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: mirroring state: enabled 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: mirroring mode: snapshot 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: mirroring global id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:01:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: mirroring primary: true 2024-04-24T16:01:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:01:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:01:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:01:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-04-24T16:01:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-24T16:01:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Discarding device blocks: 4096/2621440 done 2024-04-24T16:01:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-24T16:01:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Filesystem UUID: e948f2c4-91ce-43fa-915c-3147fc32ee67 2024-04-24T16:01:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Superblock backups stored on blocks: 2024-04-24T16:01:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-24T16:01:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: 2024-04-24T16:01:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Allocating group tables: 0/80 done 2024-04-24T16:01:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Writing inode tables: 0/80 done 2024-04-24T16:01:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Creating journal (16384 blocks): done 2024-04-24T16:01:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-24T16:01:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: 2024-04-24T16:01:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mkdir test-alternate-primary 2024-04-24T16:01:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wget https://download.ceph.com/qa/linux-5.4.tar.gz 2024-04-24T16:01:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:--2024-04-24 16:01:11-- https://download.ceph.com/qa/linux-5.4.tar.gz 2024-04-24T16:01:11.795 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-24T16:01:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:443... connected. 2024-04-24T16:01:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-24T16:01:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:Length: 172616875 (165M) [application/octet-stream] 2024-04-24T16:01:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:Saving to: ‘linux-5.4.tar.gz’ 2024-04-24T16:01:11.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2024-04-24T16:01:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 0K .......... .......... .......... .......... .......... 0% 1.35M 2m2s 2024-04-24T16:01:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50K .......... .......... .......... .......... .......... 0% 1.38M 2m0s 2024-04-24T16:01:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 100K .......... .......... .......... .......... .......... 0% 83.5M 81s 2024-04-24T16:01:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 150K .......... .......... .......... .......... .......... 0% 1.39M 90s 2024-04-24T16:01:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 200K .......... .......... .......... .......... .......... 0% 156M 72s 2024-04-24T16:01:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 250K .......... .......... .......... .......... .......... 0% 109M 60s 2024-04-24T16:01:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 300K .......... .......... .......... .......... .......... 0% 70.7M 52s 2024-04-24T16:01:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 350K .......... .......... .......... .......... .......... 0% 94.2M 46s 2024-04-24T16:01:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 400K .......... .......... .......... .......... .......... 0% 1.45M 53s 2024-04-24T16:01:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 450K .......... .......... .......... .......... .......... 0% 115M 48s 2024-04-24T16:01:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 500K .......... .......... .......... .......... .......... 0% 110M 44s 2024-04-24T16:01:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 550K .......... .......... .......... .......... .......... 0% 48.7M 41s 2024-04-24T16:01:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 600K .......... .......... .......... .......... .......... 0% 113M 37s 2024-04-24T16:01:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 650K .......... .......... .......... .......... .......... 0% 107M 35s 2024-04-24T16:01:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 700K .......... .......... .......... .......... .......... 0% 122M 33s 2024-04-24T16:01:12.133 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 750K .......... .......... .......... .......... .......... 0% 29.2M 31s 2024-04-24T16:01:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 800K .......... .......... .......... .......... .......... 0% 1.48M 36s 2024-04-24T16:01:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 850K .......... .......... .......... .......... .......... 0% 306M 34s 2024-04-24T16:01:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 900K .......... .......... .......... .......... .......... 0% 181M 32s 2024-04-24T16:01:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 950K .......... .......... .......... .......... .......... 0% 219M 30s 2024-04-24T16:01:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1000K .......... .......... .......... .......... .......... 0% 183M 29s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1050K .......... .......... .......... .......... .......... 0% 170M 28s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1100K .......... .......... .......... .......... .......... 0% 71.8M 27s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1150K .......... .......... .......... .......... .......... 0% 46.4M 26s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1200K .......... .......... .......... .......... .......... 0% 102M 25s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1250K .......... .......... .......... .......... .......... 0% 31.6M 24s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1300K .......... .......... .......... .......... .......... 0% 52.9M 23s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1350K .......... .......... .......... .......... .......... 0% 43.1M 22s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1400K .......... .......... .......... .......... .......... 0% 38.6M 22s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1450K .......... .......... .......... .......... .......... 0% 32.6M 21s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1500K .......... .......... .......... .......... .......... 0% 14.3M 21s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1550K .......... .......... .......... .......... .......... 0% 33.5M 20s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1600K .......... .......... .......... .......... .......... 0% 47.4M 20s 2024-04-24T16:01:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1650K .......... .......... .......... .......... .......... 1% 42.7M 19s 2024-04-24T16:01:12.212 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1700K .......... .......... .......... .......... .......... 1% 2.50M 21s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1750K .......... .......... .......... .......... .......... 1% 109M 20s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1800K .......... .......... .......... .......... .......... 1% 118M 20s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1850K .......... .......... .......... .......... .......... 1% 98.0M 19s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1900K .......... .......... .......... .......... .......... 1% 115M 19s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 1950K .......... .......... .......... .......... .......... 1% 44.1M 18s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2000K .......... .......... .......... .......... .......... 1% 94.3M 18s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2050K .......... .......... .......... .......... .......... 1% 41.4M 18s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2100K .......... .......... .......... .......... .......... 1% 78.7M 17s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2150K .......... .......... .......... .......... .......... 1% 22.9M 17s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2200K .......... .......... .......... .......... .......... 1% 40.6M 17s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2250K .......... .......... .......... .......... .......... 1% 28.7M 16s 2024-04-24T16:01:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2300K .......... .......... .......... .......... .......... 1% 46.9M 16s 2024-04-24T16:01:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2350K .......... .......... .......... .......... .......... 1% 13.7M 16s 2024-04-24T16:01:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2400K .......... .......... .......... .......... .......... 1% 49.3M 16s 2024-04-24T16:01:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2450K .......... .......... .......... .......... .......... 1% 48.8M 16s 2024-04-24T16:01:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2500K .......... .......... .......... .......... .......... 1% 23.0M 15s 2024-04-24T16:01:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2550K .......... .......... .......... .......... .......... 1% 28.4M 15s 2024-04-24T16:01:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2600K .......... .......... .......... .......... .......... 1% 50.1M 15s 2024-04-24T16:01:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2650K .......... .......... .......... .......... .......... 1% 33.2M 15s 2024-04-24T16:01:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2700K .......... .......... .......... .......... .......... 1% 19.6M 15s 2024-04-24T16:01:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2750K .......... .......... .......... .......... .......... 1% 27.2M 14s 2024-04-24T16:01:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2800K .......... .......... .......... .......... .......... 1% 24.0M 14s 2024-04-24T16:01:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2850K .......... .......... .......... .......... .......... 1% 80.8M 14s 2024-04-24T16:01:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2900K .......... .......... .......... .......... .......... 1% 25.3M 14s 2024-04-24T16:01:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2950K .......... .......... .......... .......... .......... 1% 35.9M 14s 2024-04-24T16:01:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3000K .......... .......... .......... .......... .......... 1% 17.7M 14s 2024-04-24T16:01:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3050K .......... .......... .......... .......... .......... 1% 30.7M 14s 2024-04-24T16:01:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3100K .......... .......... .......... .......... .......... 1% 79.9M 13s 2024-04-24T16:01:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3150K .......... .......... .......... .......... .......... 1% 20.7M 13s 2024-04-24T16:01:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3200K .......... .......... .......... .......... .......... 1% 34.7M 13s 2024-04-24T16:01:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3250K .......... .......... .......... .......... .......... 1% 23.6M 13s 2024-04-24T16:01:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3300K .......... .......... .......... .......... .......... 1% 19.9M 13s 2024-04-24T16:01:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3350K .......... .......... .......... .......... .......... 2% 113M 13s 2024-04-24T16:01:12.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3400K .......... .......... .......... .......... .......... 2% 25.2M 13s 2024-04-24T16:01:12.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3450K .......... .......... .......... .......... .......... 2% 35.9M 13s 2024-04-24T16:01:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3500K .......... .......... .......... .......... .......... 2% 24.5M 13s 2024-04-24T16:01:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3550K .......... .......... .......... .......... .......... 2% 24.4M 12s 2024-04-24T16:01:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3600K .......... .......... .......... .......... .......... 2% 20.6M 12s 2024-04-24T16:01:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3650K .......... .......... .......... .......... .......... 2% 53.4M 12s 2024-04-24T16:01:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3700K .......... .......... .......... .......... .......... 2% 50.6M 12s 2024-04-24T16:01:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3750K .......... .......... .......... .......... .......... 2% 30.3M 12s 2024-04-24T16:01:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3800K .......... .......... .......... .......... .......... 2% 20.1M 12s 2024-04-24T16:01:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3850K .......... .......... .......... .......... .......... 2% 54.4M 12s 2024-04-24T16:01:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3900K .......... .......... .......... .......... .......... 2% 21.7M 12s 2024-04-24T16:01:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 3950K .......... .......... .......... .......... .......... 2% 20.7M 12s 2024-04-24T16:01:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4000K .......... .......... .......... .......... .......... 2% 42.2M 12s 2024-04-24T16:01:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4050K .......... .......... .......... .......... .......... 2% 23.9M 12s 2024-04-24T16:01:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4100K .......... .......... .......... .......... .......... 2% 38.7M 11s 2024-04-24T16:01:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4150K .......... .......... .......... .......... .......... 2% 27.5M 11s 2024-04-24T16:01:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4200K .......... .......... .......... .......... .......... 2% 24.7M 11s 2024-04-24T16:01:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4250K .......... .......... .......... .......... .......... 2% 29.2M 11s 2024-04-24T16:01:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4300K .......... .......... .......... .......... .......... 2% 25.9M 11s 2024-04-24T16:01:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4350K .......... .......... .......... .......... .......... 2% 25.9M 11s 2024-04-24T16:01:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4400K .......... .......... .......... .......... .......... 2% 49.6M 11s 2024-04-24T16:01:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4450K .......... .......... .......... .......... .......... 2% 29.8M 11s 2024-04-24T16:01:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4500K .......... .......... .......... .......... .......... 2% 31.1M 11s 2024-04-24T16:01:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4550K .......... .......... .......... .......... .......... 2% 22.2M 11s 2024-04-24T16:01:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4600K .......... .......... .......... .......... .......... 2% 46.3M 11s 2024-04-24T16:01:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4650K .......... .......... .......... .......... .......... 2% 26.9M 11s 2024-04-24T16:01:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4700K .......... .......... .......... .......... .......... 2% 26.2M 11s 2024-04-24T16:01:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4750K .......... .......... .......... .......... .......... 2% 23.9M 11s 2024-04-24T16:01:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4800K .......... .......... .......... .......... .......... 2% 24.1M 11s 2024-04-24T16:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4850K .......... .......... .......... .......... .......... 2% 27.0M 11s 2024-04-24T16:01:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4900K .......... .......... .......... .......... .......... 2% 44.5M 10s 2024-04-24T16:01:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 4950K .......... .......... .......... .......... .......... 2% 30.5M 10s 2024-04-24T16:01:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5000K .......... .......... .......... .......... .......... 2% 24.8M 10s 2024-04-24T16:01:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5050K .......... .......... .......... .......... .......... 3% 38.3M 10s 2024-04-24T16:01:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5100K .......... .......... .......... .......... .......... 3% 33.5M 10s 2024-04-24T16:01:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5150K .......... .......... .......... .......... .......... 3% 25.3M 10s 2024-04-24T16:01:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5200K .......... .......... .......... .......... .......... 3% 22.2M 10s 2024-04-24T16:01:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5250K .......... .......... .......... .......... .......... 3% 33.1M 10s 2024-04-24T16:01:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5300K .......... .......... .......... .......... .......... 3% 27.6M 10s 2024-04-24T16:01:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5350K .......... .......... .......... .......... .......... 3% 23.5M 10s 2024-04-24T16:01:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5400K .......... .......... .......... .......... .......... 3% 47.6M 10s 2024-04-24T16:01:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5450K .......... .......... .......... .......... .......... 3% 25.0M 10s 2024-04-24T16:01:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5500K .......... .......... .......... .......... .......... 3% 24.9M 10s 2024-04-24T16:01:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5550K .......... .......... .......... .......... .......... 3% 31.1M 10s 2024-04-24T16:01:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5600K .......... .......... .......... .......... .......... 3% 36.3M 10s 2024-04-24T16:01:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5650K .......... .......... .......... .......... .......... 3% 24.9M 10s 2024-04-24T16:01:12.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5700K .......... .......... .......... .......... .......... 3% 31.1M 10s 2024-04-24T16:01:12.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5750K .......... .......... .......... .......... .......... 3% 31.0M 10s 2024-04-24T16:01:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5800K .......... .......... .......... .......... .......... 3% 22.2M 10s 2024-04-24T16:01:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5850K .......... .......... .......... .......... .......... 3% 28.4M 10s 2024-04-24T16:01:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5900K .......... .......... .......... .......... .......... 3% 51.1M 10s 2024-04-24T16:01:12.334 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 5950K .......... .......... .......... .......... .......... 3% 18.7M 10s 2024-04-24T16:01:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6000K .......... .......... .......... .......... .......... 3% 28.9M 10s 2024-04-24T16:01:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6050K .......... .......... .......... .......... .......... 3% 47.7M 9s 2024-04-24T16:01:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6100K .......... .......... .......... .......... .......... 3% 12.2M 10s 2024-04-24T16:01:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6150K .......... .......... .......... .......... .......... 3% 51.6M 9s 2024-04-24T16:01:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6200K .......... .......... .......... .......... .......... 3% 44.0M 9s 2024-04-24T16:01:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6250K .......... .......... .......... .......... .......... 3% 44.0M 9s 2024-04-24T16:01:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6300K .......... .......... .......... .......... .......... 3% 20.3M 9s 2024-04-24T16:01:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6350K .......... .......... .......... .......... .......... 3% 46.8M 9s 2024-04-24T16:01:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6400K .......... .......... .......... .......... .......... 3% 39.4M 9s 2024-04-24T16:01:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6450K .......... .......... .......... .......... .......... 3% 20.2M 9s 2024-04-24T16:01:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6500K .......... .......... .......... .......... .......... 3% 44.1M 9s 2024-04-24T16:01:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6550K .......... .......... .......... .......... .......... 3% 31.1M 9s 2024-04-24T16:01:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6600K .......... .......... .......... .......... .......... 3% 27.4M 9s 2024-04-24T16:01:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6650K .......... .......... .......... .......... .......... 3% 35.1M 9s 2024-04-24T16:01:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6700K .......... .......... .......... .......... .......... 4% 33.0M 9s 2024-04-24T16:01:12.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6750K .......... .......... .......... .......... .......... 4% 21.6M 9s 2024-04-24T16:01:12.363 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6800K .......... .......... .......... .......... .......... 4% 39.1M 9s 2024-04-24T16:01:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6850K .......... .......... .......... .......... .......... 4% 21.0M 9s 2024-04-24T16:01:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6900K .......... .......... .......... .......... .......... 4% 54.5M 9s 2024-04-24T16:01:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 6950K .......... .......... .......... .......... .......... 4% 29.5M 9s 2024-04-24T16:01:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7000K .......... .......... .......... .......... .......... 4% 26.2M 9s 2024-04-24T16:01:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7050K .......... .......... .......... .......... .......... 4% 27.8M 9s 2024-04-24T16:01:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7100K .......... .......... .......... .......... .......... 4% 23.7M 9s 2024-04-24T16:01:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7150K .......... .......... .......... .......... .......... 4% 21.7M 9s 2024-04-24T16:01:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7200K .......... .......... .......... .......... .......... 4% 58.9M 9s 2024-04-24T16:01:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7250K .......... .......... .......... .......... .......... 4% 28.0M 9s 2024-04-24T16:01:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7300K .......... .......... .......... .......... .......... 4% 27.3M 9s 2024-04-24T16:01:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7350K .......... .......... .......... .......... .......... 4% 27.0M 9s 2024-04-24T16:01:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7400K .......... .......... .......... .......... .......... 4% 45.3M 9s 2024-04-24T16:01:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7450K .......... .......... .......... .......... .......... 4% 26.6M 9s 2024-04-24T16:01:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7500K .......... .......... .......... .......... .......... 4% 26.3M 9s 2024-04-24T16:01:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7550K .......... .......... .......... .......... .......... 4% 24.4M 9s 2024-04-24T16:01:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7600K .......... .......... .......... .......... .......... 4% 25.5M 9s 2024-04-24T16:01:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7650K .......... .......... .......... .......... .......... 4% 28.6M 9s 2024-04-24T16:01:12.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7700K .......... .......... .......... .......... .......... 4% 50.6M 9s 2024-04-24T16:01:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7750K .......... .......... .......... .......... .......... 4% 24.1M 9s 2024-04-24T16:01:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7800K .......... .......... .......... .......... .......... 4% 25.3M 9s 2024-04-24T16:01:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7850K .......... .......... .......... .......... .......... 4% 36.5M 8s 2024-04-24T16:01:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7900K .......... .......... .......... .......... .......... 4% 48.0M 8s 2024-04-24T16:01:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 7950K .......... .......... .......... .......... .......... 4% 22.9M 8s 2024-04-24T16:01:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8000K .......... .......... .......... .......... .......... 4% 21.2M 8s 2024-04-24T16:01:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8050K .......... .......... .......... .......... .......... 4% 45.6M 8s 2024-04-24T16:01:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8100K .......... .......... .......... .......... .......... 4% 25.8M 8s 2024-04-24T16:01:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8150K .......... .......... .......... .......... .......... 4% 22.0M 8s 2024-04-24T16:01:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8200K .......... .......... .......... .......... .......... 4% 60.0M 8s 2024-04-24T16:01:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8250K .......... .......... .......... .......... .......... 4% 20.9M 8s 2024-04-24T16:01:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8300K .......... .......... .......... .......... .......... 4% 36.5M 8s 2024-04-24T16:01:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8350K .......... .......... .......... .......... .......... 4% 21.8M 8s 2024-04-24T16:01:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8400K .......... .......... .......... .......... .......... 5% 29.6M 8s 2024-04-24T16:01:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8450K .......... .......... .......... .......... .......... 5% 49.7M 8s 2024-04-24T16:01:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8500K .......... .......... .......... .......... .......... 5% 27.1M 8s 2024-04-24T16:01:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8550K .......... .......... .......... .......... .......... 5% 31.3M 8s 2024-04-24T16:01:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8600K .......... .......... .......... .......... .......... 5% 21.9M 8s 2024-04-24T16:01:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8650K .......... .......... .......... .......... .......... 5% 22.7M 8s 2024-04-24T16:01:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8700K .......... .......... .......... .......... .......... 5% 79.2M 8s 2024-04-24T16:01:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8750K .......... .......... .......... .......... .......... 5% 22.9M 8s 2024-04-24T16:01:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8800K .......... .......... .......... .......... .......... 5% 26.1M 8s 2024-04-24T16:01:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8850K .......... .......... .......... .......... .......... 5% 28.1M 8s 2024-04-24T16:01:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8900K .......... .......... .......... .......... .......... 5% 34.8M 8s 2024-04-24T16:01:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 8950K .......... .......... .......... .......... .......... 5% 37.2M 8s 2024-04-24T16:01:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9000K .......... .......... .......... .......... .......... 5% 22.3M 8s 2024-04-24T16:01:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9050K .......... .......... .......... .......... .......... 5% 29.9M 8s 2024-04-24T16:01:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9100K .......... .......... .......... .......... .......... 5% 45.6M 8s 2024-04-24T16:01:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9150K .......... .......... .......... .......... .......... 5% 23.3M 8s 2024-04-24T16:01:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9200K .......... .......... .......... .......... .......... 5% 20.9M 8s 2024-04-24T16:01:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9250K .......... .......... .......... .......... .......... 5% 77.7M 8s 2024-04-24T16:01:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9300K .......... .......... .......... .......... .......... 5% 22.0M 8s 2024-04-24T16:01:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9350K .......... .......... .......... .......... .......... 5% 32.1M 8s 2024-04-24T16:01:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9400K .......... .......... .......... .......... .......... 5% 39.1M 8s 2024-04-24T16:01:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9450K .......... .......... .......... .......... .......... 5% 28.4M 8s 2024-04-24T16:01:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9500K .......... .......... .......... .......... .......... 5% 26.3M 8s 2024-04-24T16:01:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9550K .......... .......... .......... .......... .......... 5% 22.8M 8s 2024-04-24T16:01:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9600K .......... .......... .......... .......... .......... 5% 27.6M 8s 2024-04-24T16:01:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9650K .......... .......... .......... .......... .......... 5% 39.6M 8s 2024-04-24T16:01:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9700K .......... .......... .......... .......... .......... 5% 36.2M 8s 2024-04-24T16:01:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9750K .......... .......... .......... .......... .......... 5% 23.0M 8s 2024-04-24T16:01:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9800K .......... .......... .......... .......... .......... 5% 37.5M 8s 2024-04-24T16:01:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9850K .......... .......... .......... .......... .......... 5% 22.5M 8s 2024-04-24T16:01:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9900K .......... .......... .......... .......... .......... 5% 26.1M 8s 2024-04-24T16:01:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 9950K .......... .......... .......... .......... .......... 5% 23.0M 8s 2024-04-24T16:01:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10000K .......... .......... .......... .......... .......... 5% 50.1M 8s 2024-04-24T16:01:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10050K .......... .......... .......... .......... .......... 5% 26.0M 8s 2024-04-24T16:01:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10100K .......... .......... .......... .......... .......... 6% 29.3M 8s 2024-04-24T16:01:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10150K .......... .......... .......... .......... .......... 6% 49.8M 8s 2024-04-24T16:01:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10200K .......... .......... .......... .......... .......... 6% 29.2M 8s 2024-04-24T16:01:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10250K .......... .......... .......... .......... .......... 6% 25.1M 8s 2024-04-24T16:01:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10300K .......... .......... .......... .......... .......... 6% 24.3M 8s 2024-04-24T16:01:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10350K .......... .......... .......... .......... .......... 6% 22.1M 8s 2024-04-24T16:01:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10400K .......... .......... .......... .......... .......... 6% 46.0M 8s 2024-04-24T16:01:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10450K .......... .......... .......... .......... .......... 6% 23.9M 8s 2024-04-24T16:01:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10500K .......... .......... .......... .......... .......... 6% 37.9M 8s 2024-04-24T16:01:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10550K .......... .......... .......... .......... .......... 6% 22.0M 8s 2024-04-24T16:01:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10600K .......... .......... .......... .......... .......... 6% 43.4M 8s 2024-04-24T16:01:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10650K .......... .......... .......... .......... .......... 6% 20.8M 8s 2024-04-24T16:01:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10700K .......... .......... .......... .......... .......... 6% 75.8M 8s 2024-04-24T16:01:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10750K .......... .......... .......... .......... .......... 6% 19.8M 8s 2024-04-24T16:01:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10800K .......... .......... .......... .......... .......... 6% 28.4M 8s 2024-04-24T16:01:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10850K .......... .......... .......... .......... .......... 6% 29.7M 7s 2024-04-24T16:01:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10900K .......... .......... .......... .......... .......... 6% 26.0M 7s 2024-04-24T16:01:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 10950K .......... .......... .......... .......... .......... 6% 62.7M 7s 2024-04-24T16:01:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11000K .......... .......... .......... .......... .......... 6% 25.0M 7s 2024-04-24T16:01:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11050K .......... .......... .......... .......... .......... 6% 35.5M 7s 2024-04-24T16:01:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11100K .......... .......... .......... .......... .......... 6% 25.2M 7s 2024-04-24T16:01:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11150K .......... .......... .......... .......... .......... 6% 21.9M 7s 2024-04-24T16:01:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11200K .......... .......... .......... .......... .......... 6% 27.9M 7s 2024-04-24T16:01:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11250K .......... .......... .......... .......... .......... 6% 34.4M 7s 2024-04-24T16:01:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11300K .......... .......... .......... .......... .......... 6% 25.0M 7s 2024-04-24T16:01:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11350K .......... .......... .......... .......... .......... 6% 44.5M 7s 2024-04-24T16:01:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11400K .......... .......... .......... .......... .......... 6% 22.8M 7s 2024-04-24T16:01:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11450K .......... .......... .......... .......... .......... 6% 29.0M 7s 2024-04-24T16:01:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11500K .......... .......... .......... .......... .......... 6% 39.1M 7s 2024-04-24T16:01:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11550K .......... .......... .......... .......... .......... 6% 26.4M 7s 2024-04-24T16:01:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11600K .......... .......... .......... .......... .......... 6% 27.3M 7s 2024-04-24T16:01:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11650K .......... .......... .......... .......... .......... 6% 40.7M 7s 2024-04-24T16:01:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11700K .......... .......... .......... .......... .......... 6% 21.4M 7s 2024-04-24T16:01:12.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11750K .......... .......... .......... .......... .......... 7% 28.4M 7s 2024-04-24T16:01:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11800K .......... .......... .......... .......... .......... 7% 27.3M 7s 2024-04-24T16:01:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11850K .......... .......... .......... .......... .......... 7% 40.4M 7s 2024-04-24T16:01:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11900K .......... .......... .......... .......... .......... 7% 24.1M 7s 2024-04-24T16:01:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 11950K .......... .......... .......... .......... .......... 7% 23.0M 7s 2024-04-24T16:01:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12000K .......... .......... .......... .......... .......... 7% 50.6M 7s 2024-04-24T16:01:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12050K .......... .......... .......... .......... .......... 7% 29.4M 7s 2024-04-24T16:01:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12100K .......... .......... .......... .......... .......... 7% 28.5M 7s 2024-04-24T16:01:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12150K .......... .......... .......... .......... .......... 7% 22.6M 7s 2024-04-24T16:01:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12200K .......... .......... .......... .......... .......... 7% 23.5M 7s 2024-04-24T16:01:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12250K .......... .......... .......... .......... .......... 7% 45.1M 7s 2024-04-24T16:01:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12300K .......... .......... .......... .......... .......... 7% 24.7M 7s 2024-04-24T16:01:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12350K .......... .......... .......... .......... .......... 7% 28.1M 7s 2024-04-24T16:01:12.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12400K .......... .......... .......... .......... .......... 7% 29.1M 7s 2024-04-24T16:01:12.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12450K .......... .......... .......... .......... .......... 7% 24.9M 7s 2024-04-24T16:01:12.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12500K .......... .......... .......... .......... .......... 7% 55.9M 7s 2024-04-24T16:01:12.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12550K .......... .......... .......... .......... .......... 7% 21.2M 7s 2024-04-24T16:01:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12600K .......... .......... .......... .......... .......... 7% 37.7M 7s 2024-04-24T16:01:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12650K .......... .......... .......... .......... .......... 7% 25.8M 7s 2024-04-24T16:01:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12700K .......... .......... .......... .......... .......... 7% 32.5M 7s 2024-04-24T16:01:12.564 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12750K .......... .......... .......... .......... .......... 7% 18.9M 7s 2024-04-24T16:01:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12800K .......... .......... .......... .......... .......... 7% 44.6M 7s 2024-04-24T16:01:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12850K .......... .......... .......... .......... .......... 7% 21.5M 7s 2024-04-24T16:01:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12900K .......... .......... .......... .......... .......... 7% 32.6M 7s 2024-04-24T16:01:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 12950K .......... .......... .......... .......... .......... 7% 32.1M 7s 2024-04-24T16:01:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13000K .......... .......... .......... .......... .......... 7% 45.5M 7s 2024-04-24T16:01:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13050K .......... .......... .......... .......... .......... 7% 24.4M 7s 2024-04-24T16:01:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13100K .......... .......... .......... .......... .......... 7% 31.4M 7s 2024-04-24T16:01:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13150K .......... .......... .......... .......... .......... 7% 25.4M 7s 2024-04-24T16:01:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13200K .......... .......... .......... .......... .......... 7% 21.6M 7s 2024-04-24T16:01:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13250K .......... .......... .......... .......... .......... 7% 24.3M 7s 2024-04-24T16:01:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13300K .......... .......... .......... .......... .......... 7% 53.2M 7s 2024-04-24T16:01:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13350K .......... .......... .......... .......... .......... 7% 23.6M 7s 2024-04-24T16:01:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13400K .......... .......... .......... .......... .......... 7% 28.5M 7s 2024-04-24T16:01:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13450K .......... .......... .......... .......... .......... 8% 30.7M 7s 2024-04-24T16:01:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13500K .......... .......... .......... .......... .......... 8% 50.4M 7s 2024-04-24T16:01:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13550K .......... .......... .......... .......... .......... 8% 22.8M 7s 2024-04-24T16:01:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13600K .......... .......... .......... .......... .......... 8% 27.2M 7s 2024-04-24T16:01:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13650K .......... .......... .......... .......... .......... 8% 28.3M 7s 2024-04-24T16:01:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13700K .......... .......... .......... .......... .......... 8% 48.3M 7s 2024-04-24T16:01:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13750K .......... .......... .......... .......... .......... 8% 18.1M 7s 2024-04-24T16:01:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13800K .......... .......... .......... .......... .......... 8% 44.7M 7s 2024-04-24T16:01:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13850K .......... .......... .......... .......... .......... 8% 32.8M 7s 2024-04-24T16:01:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13900K .......... .......... .......... .......... .......... 8% 27.3M 7s 2024-04-24T16:01:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 13950K .......... .......... .......... .......... .......... 8% 20.4M 7s 2024-04-24T16:01:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14000K .......... .......... .......... .......... .......... 8% 25.3M 7s 2024-04-24T16:01:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14050K .......... .......... .......... .......... .......... 8% 79.7M 7s 2024-04-24T16:01:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14100K .......... .......... .......... .......... .......... 8% 21.4M 7s 2024-04-24T16:01:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14150K .......... .......... .......... .......... .......... 8% 32.2M 7s 2024-04-24T16:01:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14200K .......... .......... .......... .......... .......... 8% 30.4M 7s 2024-04-24T16:01:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14250K .......... .......... .......... .......... .......... 8% 48.8M 7s 2024-04-24T16:01:12.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14300K .......... .......... .......... .......... .......... 8% 25.1M 7s 2024-04-24T16:01:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14350K .......... .......... .......... .......... .......... 8% 23.5M 7s 2024-04-24T16:01:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14400K .......... .......... .......... .......... .......... 8% 22.1M 7s 2024-04-24T16:01:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14450K .......... .......... .......... .......... .......... 8% 53.4M 7s 2024-04-24T16:01:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14500K .......... .......... .......... .......... .......... 8% 22.9M 7s 2024-04-24T16:01:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14550K .......... .......... .......... .......... .......... 8% 47.2M 7s 2024-04-24T16:01:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14600K .......... .......... .......... .......... .......... 8% 23.4M 7s 2024-04-24T16:01:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14650K .......... .......... .......... .......... .......... 8% 30.4M 7s 2024-04-24T16:01:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14700K .......... .......... .......... .......... .......... 8% 28.8M 7s 2024-04-24T16:01:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14750K .......... .......... .......... .......... .......... 8% 23.6M 7s 2024-04-24T16:01:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14800K .......... .......... .......... .......... .......... 8% 48.0M 7s 2024-04-24T16:01:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14850K .......... .......... .......... .......... .......... 8% 25.3M 7s 2024-04-24T16:01:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14900K .......... .......... .......... .......... .......... 8% 23.3M 7s 2024-04-24T16:01:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 14950K .......... .......... .......... .......... .......... 8% 36.3M 7s 2024-04-24T16:01:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15000K .......... .......... .......... .......... .......... 8% 24.4M 7s 2024-04-24T16:01:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15050K .......... .......... .......... .......... .......... 8% 73.7M 7s 2024-04-24T16:01:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15100K .......... .......... .......... .......... .......... 8% 19.4M 7s 2024-04-24T16:01:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15150K .......... .......... .......... .......... .......... 9% 26.6M 7s 2024-04-24T16:01:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15200K .......... .......... .......... .......... .......... 9% 30.3M 7s 2024-04-24T16:01:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15250K .......... .......... .......... .......... .......... 9% 27.4M 7s 2024-04-24T16:01:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15300K .......... .......... .......... .......... .......... 9% 40.0M 7s 2024-04-24T16:01:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15350K .......... .......... .......... .......... .......... 9% 29.1M 7s 2024-04-24T16:01:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15400K .......... .......... .......... .......... .......... 9% 35.3M 7s 2024-04-24T16:01:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15450K .......... .......... .......... .......... .......... 9% 34.5M 7s 2024-04-24T16:01:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15500K .......... .......... .......... .......... .......... 9% 25.0M 7s 2024-04-24T16:01:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15550K .......... .......... .......... .......... .......... 9% 16.4M 7s 2024-04-24T16:01:12.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15600K .......... .......... .......... .......... .......... 9% 41.6M 7s 2024-04-24T16:01:12.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15650K .......... .......... .......... .......... .......... 9% 26.0M 7s 2024-04-24T16:01:12.664 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15700K .......... .......... .......... .......... .......... 9% 36.6M 7s 2024-04-24T16:01:12.666 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15750K .......... .......... .......... .......... .......... 9% 23.3M 7s 2024-04-24T16:01:12.667 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15800K .......... .......... .......... .......... .......... 9% 45.5M 7s 2024-04-24T16:01:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15850K .......... .......... .......... .......... .......... 9% 27.5M 7s 2024-04-24T16:01:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15900K .......... .......... .......... .......... .......... 9% 38.9M 7s 2024-04-24T16:01:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 15950K .......... .......... .......... .......... .......... 9% 22.7M 7s 2024-04-24T16:01:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16000K .......... .......... .......... .......... .......... 9% 25.0M 7s 2024-04-24T16:01:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16050K .......... .......... .......... .......... .......... 9% 24.9M 7s 2024-04-24T16:01:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16100K .......... .......... .......... .......... .......... 9% 49.1M 7s 2024-04-24T16:01:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16150K .......... .......... .......... .......... .......... 9% 23.2M 7s 2024-04-24T16:01:12.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16200K .......... .......... .......... .......... .......... 9% 35.6M 7s 2024-04-24T16:01:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16250K .......... .......... .......... .......... .......... 9% 24.4M 7s 2024-04-24T16:01:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16300K .......... .......... .......... .......... .......... 9% 65.2M 7s 2024-04-24T16:01:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16350K .......... .......... .......... .......... .......... 9% 21.2M 7s 2024-04-24T16:01:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16400K .......... .......... .......... .......... .......... 9% 26.3M 7s 2024-04-24T16:01:12.690 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16450K .......... .......... .......... .......... .......... 9% 21.8M 7s 2024-04-24T16:01:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16500K .......... .......... .......... .......... .......... 9% 39.7M 7s 2024-04-24T16:01:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16550K .......... .......... .......... .......... .......... 9% 21.8M 7s 2024-04-24T16:01:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16600K .......... .......... .......... .......... .......... 9% 73.2M 6s 2024-04-24T16:01:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16650K .......... .......... .......... .......... .......... 9% 24.9M 6s 2024-04-24T16:01:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16700K .......... .......... .......... .......... .......... 9% 44.6M 6s 2024-04-24T16:01:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16750K .......... .......... .......... .......... .......... 9% 16.0M 6s 2024-04-24T16:01:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16800K .......... .......... .......... .......... .......... 9% 32.9M 6s 2024-04-24T16:01:12.702 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16850K .......... .......... .......... .......... .......... 10% 38.3M 6s 2024-04-24T16:01:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16900K .......... .......... .......... .......... .......... 10% 23.7M 6s 2024-04-24T16:01:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 16950K .......... .......... .......... .......... .......... 10% 33.2M 6s 2024-04-24T16:01:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17000K .......... .......... .......... .......... .......... 10% 41.7M 6s 2024-04-24T16:01:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17050K .......... .......... .......... .......... .......... 10% 38.2M 6s 2024-04-24T16:01:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17100K .......... .......... .......... .......... .......... 10% 22.2M 6s 2024-04-24T16:01:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17150K .......... .......... .......... .......... .......... 10% 20.8M 6s 2024-04-24T16:01:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17200K .......... .......... .......... .......... .......... 10% 25.0M 6s 2024-04-24T16:01:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17250K .......... .......... .......... .......... .......... 10% 49.9M 6s 2024-04-24T16:01:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17300K .......... .......... .......... .......... .......... 10% 25.0M 6s 2024-04-24T16:01:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17350K .......... .......... .......... .......... .......... 10% 47.0M 6s 2024-04-24T16:01:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17400K .......... .......... .......... .......... .......... 10% 25.0M 6s 2024-04-24T16:01:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17450K .......... .......... .......... .......... .......... 10% 34.5M 6s 2024-04-24T16:01:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17500K .......... .......... .......... .......... .......... 10% 33.3M 6s 2024-04-24T16:01:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17550K .......... .......... .......... .......... .......... 10% 21.7M 6s 2024-04-24T16:01:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17600K .......... .......... .......... .......... .......... 10% 32.6M 6s 2024-04-24T16:01:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17650K .......... .......... .......... .......... .......... 10% 34.7M 6s 2024-04-24T16:01:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17700K .......... .......... .......... .......... .......... 10% 28.4M 6s 2024-04-24T16:01:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17750K .......... .......... .......... .......... .......... 10% 23.1M 6s 2024-04-24T16:01:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17800K .......... .......... .......... .......... .......... 10% 25.2M 6s 2024-04-24T16:01:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17850K .......... .......... .......... .......... .......... 10% 42.2M 6s 2024-04-24T16:01:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17900K .......... .......... .......... .......... .......... 10% 43.1M 6s 2024-04-24T16:01:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 17950K .......... .......... .......... .......... .......... 10% 19.1M 6s 2024-04-24T16:01:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18000K .......... .......... .......... .......... .......... 10% 40.4M 6s 2024-04-24T16:01:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18050K .......... .......... .......... .......... .......... 10% 27.6M 6s 2024-04-24T16:01:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18100K .......... .......... .......... .......... .......... 10% 27.5M 6s 2024-04-24T16:01:12.746 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18150K .......... .......... .......... .......... .......... 10% 31.5M 6s 2024-04-24T16:01:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18200K .......... .......... .......... .......... .......... 10% 22.8M 6s 2024-04-24T16:01:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18250K .......... .......... .......... .......... .......... 10% 35.2M 6s 2024-04-24T16:01:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18300K .......... .......... .......... .......... .......... 10% 27.4M 6s 2024-04-24T16:01:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18350K .......... .......... .......... .......... .......... 10% 24.1M 6s 2024-04-24T16:01:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18400K .......... .......... .......... .......... .......... 10% 31.0M 6s 2024-04-24T16:01:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18450K .......... .......... .......... .......... .......... 10% 30.3M 6s 2024-04-24T16:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18500K .......... .......... .......... .......... .......... 11% 42.9M 6s 2024-04-24T16:01:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18550K .......... .......... .......... .......... .......... 11% 21.6M 6s 2024-04-24T16:01:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18600K .......... .......... .......... .......... .......... 11% 39.3M 6s 2024-04-24T16:01:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18650K .......... .......... .......... .......... .......... 11% 33.8M 6s 2024-04-24T16:01:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18700K .......... .......... .......... .......... .......... 11% 24.5M 6s 2024-04-24T16:01:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18750K .......... .......... .......... .......... .......... 11% 23.6M 6s 2024-04-24T16:01:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18800K .......... .......... .......... .......... .......... 11% 27.3M 6s 2024-04-24T16:01:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18850K .......... .......... .......... .......... .......... 11% 41.0M 6s 2024-04-24T16:01:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18900K .......... .......... .......... .......... .......... 11% 32.4M 6s 2024-04-24T16:01:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 18950K .......... .......... .......... .......... .......... 11% 25.3M 6s 2024-04-24T16:01:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19000K .......... .......... .......... .......... .......... 11% 29.6M 6s 2024-04-24T16:01:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19050K .......... .......... .......... .......... .......... 11% 27.1M 6s 2024-04-24T16:01:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19100K .......... .......... .......... .......... .......... 11% 39.1M 6s 2024-04-24T16:01:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19150K .......... .......... .......... .......... .......... 11% 22.3M 6s 2024-04-24T16:01:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19200K .......... .......... .......... .......... .......... 11% 21.8M 6s 2024-04-24T16:01:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19250K .......... .......... .......... .......... .......... 11% 59.3M 6s 2024-04-24T16:01:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19300K .......... .......... .......... .......... .......... 11% 30.9M 6s 2024-04-24T16:01:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19350K .......... .......... .......... .......... .......... 11% 34.7M 6s 2024-04-24T16:01:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19400K .......... .......... .......... .......... .......... 11% 23.1M 6s 2024-04-24T16:01:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19450K .......... .......... .......... .......... .......... 11% 35.2M 6s 2024-04-24T16:01:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19500K .......... .......... .......... .......... .......... 11% 34.5M 6s 2024-04-24T16:01:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19550K .......... .......... .......... .......... .......... 11% 22.9M 6s 2024-04-24T16:01:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19600K .......... .......... .......... .......... .......... 11% 22.0M 6s 2024-04-24T16:01:12.796 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19650K .......... .......... .......... .......... .......... 11% 37.6M 6s 2024-04-24T16:01:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19700K .......... .......... .......... .......... .......... 11% 20.8M 6s 2024-04-24T16:01:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19750K .......... .......... .......... .......... .......... 11% 43.2M 6s 2024-04-24T16:01:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19800K .......... .......... .......... .......... .......... 11% 22.6M 6s 2024-04-24T16:01:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19850K .......... .......... .......... .......... .......... 11% 89.1M 6s 2024-04-24T16:01:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19900K .......... .......... .......... .......... .......... 11% 19.8M 6s 2024-04-24T16:01:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 19950K .......... .......... .......... .......... .......... 11% 22.1M 6s 2024-04-24T16:01:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20000K .......... .......... .......... .......... .......... 11% 47.6M 6s 2024-04-24T16:01:12.811 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20050K .......... .......... .......... .......... .......... 11% 24.3M 6s 2024-04-24T16:01:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20100K .......... .......... .......... .......... .......... 11% 21.8M 6s 2024-04-24T16:01:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20150K .......... .......... .......... .......... .......... 11% 92.9M 6s 2024-04-24T16:01:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20200K .......... .......... .......... .......... .......... 12% 27.3M 6s 2024-04-24T16:01:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20250K .......... .......... .......... .......... .......... 12% 21.3M 6s 2024-04-24T16:01:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20300K .......... .......... .......... .......... .......... 12% 48.6M 6s 2024-04-24T16:01:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20350K .......... .......... .......... .......... .......... 12% 23.4M 6s 2024-04-24T16:01:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20400K .......... .......... .......... .......... .......... 12% 33.4M 6s 2024-04-24T16:01:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20450K .......... .......... .......... .......... .......... 12% 24.7M 6s 2024-04-24T16:01:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20500K .......... .......... .......... .......... .......... 12% 19.9M 6s 2024-04-24T16:01:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20550K .......... .......... .......... .......... .......... 12% 75.0M 6s 2024-04-24T16:01:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20600K .......... .......... .......... .......... .......... 12% 21.7M 6s 2024-04-24T16:01:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20650K .......... .......... .......... .......... .......... 12% 48.0M 6s 2024-04-24T16:01:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20700K .......... .......... .......... .......... .......... 12% 18.6M 6s 2024-04-24T16:01:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20750K .......... .......... .......... .......... .......... 12% 29.3M 6s 2024-04-24T16:01:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20800K .......... .......... .......... .......... .......... 12% 20.1M 6s 2024-04-24T16:01:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20850K .......... .......... .......... .......... .......... 12% 36.9M 6s 2024-04-24T16:01:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20900K .......... .......... .......... .......... .......... 12% 55.1M 6s 2024-04-24T16:01:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 20950K .......... .......... .......... .......... .......... 12% 23.6M 6s 2024-04-24T16:01:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21000K .......... .......... .......... .......... .......... 12% 42.3M 6s 2024-04-24T16:01:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21050K .......... .......... .......... .......... .......... 12% 21.9M 6s 2024-04-24T16:01:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21100K .......... .......... .......... .......... .......... 12% 26.4M 6s 2024-04-24T16:01:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21150K .......... .......... .......... .......... .......... 12% 20.8M 6s 2024-04-24T16:01:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21200K .......... .......... .......... .......... .......... 12% 54.9M 6s 2024-04-24T16:01:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21250K .......... .......... .......... .......... .......... 12% 34.5M 6s 2024-04-24T16:01:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21300K .......... .......... .......... .......... .......... 12% 24.7M 6s 2024-04-24T16:01:12.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21350K .......... .......... .......... .......... .......... 12% 32.8M 6s 2024-04-24T16:01:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21400K .......... .......... .......... .......... .......... 12% 36.5M 6s 2024-04-24T16:01:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21450K .......... .......... .......... .......... .......... 12% 19.5M 6s 2024-04-24T16:01:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21500K .......... .......... .......... .......... .......... 12% 24.8M 6s 2024-04-24T16:01:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21550K .......... .......... .......... .......... .......... 12% 34.2M 6s 2024-04-24T16:01:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21600K .......... .......... .......... .......... .......... 12% 36.4M 6s 2024-04-24T16:01:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21650K .......... .......... .......... .......... .......... 12% 17.1M 6s 2024-04-24T16:01:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21700K .......... .......... .......... .......... .......... 12% 57.0M 6s 2024-04-24T16:01:12.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21750K .......... .......... .......... .......... .......... 12% 34.5M 6s 2024-04-24T16:01:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21800K .......... .......... .......... .......... .......... 12% 29.6M 6s 2024-04-24T16:01:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21850K .......... .......... .......... .......... .......... 12% 23.8M 6s 2024-04-24T16:01:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21900K .......... .......... .......... .......... .......... 13% 31.9M 6s 2024-04-24T16:01:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 21950K .......... .......... .......... .......... .......... 13% 23.9M 6s 2024-04-24T16:01:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22000K .......... .......... .......... .......... .......... 13% 33.4M 6s 2024-04-24T16:01:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22050K .......... .......... .......... .......... .......... 13% 28.8M 6s 2024-04-24T16:01:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22100K .......... .......... .......... .......... .......... 13% 32.3M 6s 2024-04-24T16:01:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22150K .......... .......... .......... .......... .......... 13% 35.1M 6s 2024-04-24T16:01:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22200K .......... .......... .......... .......... .......... 13% 22.4M 6s 2024-04-24T16:01:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22250K .......... .......... .......... .......... .......... 13% 39.7M 6s 2024-04-24T16:01:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22300K .......... .......... .......... .......... .......... 13% 20.5M 6s 2024-04-24T16:01:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22350K .......... .......... .......... .......... .......... 13% 31.4M 6s 2024-04-24T16:01:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22400K .......... .......... .......... .......... .......... 13% 23.7M 6s 2024-04-24T16:01:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22450K .......... .......... .......... .......... .......... 13% 47.8M 6s 2024-04-24T16:01:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22500K .......... .......... .......... .......... .......... 13% 22.8M 6s 2024-04-24T16:01:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22550K .......... .......... .......... .......... .......... 13% 43.5M 6s 2024-04-24T16:01:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22600K .......... .......... .......... .......... .......... 13% 20.9M 6s 2024-04-24T16:01:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22650K .......... .......... .......... .......... .......... 13% 66.3M 6s 2024-04-24T16:01:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22700K .......... .......... .......... .......... .......... 13% 23.5M 6s 2024-04-24T16:01:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22750K .......... .......... .......... .......... .......... 13% 24.2M 6s 2024-04-24T16:01:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22800K .......... .......... .......... .......... .......... 13% 30.3M 6s 2024-04-24T16:01:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22850K .......... .......... .......... .......... .......... 13% 19.5M 6s 2024-04-24T16:01:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22900K .......... .......... .......... .......... .......... 13% 38.8M 6s 2024-04-24T16:01:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 22950K .......... .......... .......... .......... .......... 13% 56.5M 6s 2024-04-24T16:01:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23000K .......... .......... .......... .......... .......... 13% 23.8M 6s 2024-04-24T16:01:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23050K .......... .......... .......... .......... .......... 13% 35.2M 6s 2024-04-24T16:01:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23100K .......... .......... .......... .......... .......... 13% 25.6M 6s 2024-04-24T16:01:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23150K .......... .......... .......... .......... .......... 13% 24.9M 6s 2024-04-24T16:01:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23200K .......... .......... .......... .......... .......... 13% 37.6M 6s 2024-04-24T16:01:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23250K .......... .......... .......... .......... .......... 13% 31.5M 6s 2024-04-24T16:01:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23300K .......... .......... .......... .......... .......... 13% 28.5M 6s 2024-04-24T16:01:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23350K .......... .......... .......... .......... .......... 13% 22.6M 6s 2024-04-24T16:01:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23400K .......... .......... .......... .......... .......... 13% 36.6M 6s 2024-04-24T16:01:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23450K .......... .......... .......... .......... .......... 13% 31.4M 6s 2024-04-24T16:01:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23500K .......... .......... .......... .......... .......... 13% 25.4M 6s 2024-04-24T16:01:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23550K .......... .......... .......... .......... .......... 14% 28.8M 6s 2024-04-24T16:01:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23600K .......... .......... .......... .......... .......... 14% 26.9M 6s 2024-04-24T16:01:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23650K .......... .......... .......... .......... .......... 14% 26.1M 6s 2024-04-24T16:01:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23700K .......... .......... .......... .......... .......... 14% 41.5M 6s 2024-04-24T16:01:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23750K .......... .......... .......... .......... .......... 14% 27.2M 6s 2024-04-24T16:01:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23800K .......... .......... .......... .......... .......... 14% 32.9M 6s 2024-04-24T16:01:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23850K .......... .......... .......... .......... .......... 14% 28.2M 6s 2024-04-24T16:01:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23900K .......... .......... .......... .......... .......... 14% 43.8M 6s 2024-04-24T16:01:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 23950K .......... .......... .......... .......... .......... 14% 14.1M 6s 2024-04-24T16:01:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24000K .......... .......... .......... .......... .......... 14% 48.1M 6s 2024-04-24T16:01:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24050K .......... .......... .......... .......... .......... 14% 29.6M 6s 2024-04-24T16:01:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24100K .......... .......... .......... .......... .......... 14% 35.9M 6s 2024-04-24T16:01:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24150K .......... .......... .......... .......... .......... 14% 24.7M 6s 2024-04-24T16:01:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24200K .......... .......... .......... .......... .......... 14% 41.5M 6s 2024-04-24T16:01:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24250K .......... .......... .......... .......... .......... 14% 28.2M 6s 2024-04-24T16:01:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24300K .......... .......... .......... .......... .......... 14% 38.1M 6s 2024-04-24T16:01:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24350K .......... .......... .......... .......... .......... 14% 21.6M 6s 2024-04-24T16:01:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24400K .......... .......... .......... .......... .......... 14% 23.3M 6s 2024-04-24T16:01:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24450K .......... .......... .......... .......... .......... 14% 56.5M 6s 2024-04-24T16:01:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24500K .......... .......... .......... .......... .......... 14% 22.8M 6s 2024-04-24T16:01:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24550K .......... .......... .......... .......... .......... 14% 25.2M 6s 2024-04-24T16:01:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24600K .......... .......... .......... .......... .......... 14% 38.0M 6s 2024-04-24T16:01:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24650K .......... .......... .......... .......... .......... 14% 26.5M 6s 2024-04-24T16:01:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24700K .......... .......... .......... .......... .......... 14% 53.1M 6s 2024-04-24T16:01:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24750K .......... .......... .......... .......... .......... 14% 21.2M 6s 2024-04-24T16:01:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24800K .......... .......... .......... .......... .......... 14% 23.9M 6s 2024-04-24T16:01:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24850K .......... .......... .......... .......... .......... 14% 37.9M 6s 2024-04-24T16:01:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24900K .......... .......... .......... .......... .......... 14% 27.0M 6s 2024-04-24T16:01:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 24950K .......... .......... .......... .......... .......... 14% 38.2M 6s 2024-04-24T16:01:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25000K .......... .......... .......... .......... .......... 14% 30.0M 6s 2024-04-24T16:01:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25050K .......... .......... .......... .......... .......... 14% 25.7M 6s 2024-04-24T16:01:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25100K .......... .......... .......... .......... .......... 14% 24.3M 6s 2024-04-24T16:01:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25150K .......... .......... .......... .......... .......... 14% 29.5M 6s 2024-04-24T16:01:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25200K .......... .......... .......... .......... .......... 14% 23.1M 6s 2024-04-24T16:01:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25250K .......... .......... .......... .......... .......... 15% 60.2M 6s 2024-04-24T16:01:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25300K .......... .......... .......... .......... .......... 15% 21.5M 6s 2024-04-24T16:01:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25350K .......... .......... .......... .......... .......... 15% 29.8M 6s 2024-04-24T16:01:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25400K .......... .......... .......... .......... .......... 15% 28.1M 6s 2024-04-24T16:01:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25450K .......... .......... .......... .......... .......... 15% 57.0M 6s 2024-04-24T16:01:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25500K .......... .......... .......... .......... .......... 15% 26.5M 6s 2024-04-24T16:01:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25550K .......... .......... .......... .......... .......... 15% 20.8M 6s 2024-04-24T16:01:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25600K .......... .......... .......... .......... .......... 15% 38.4M 6s 2024-04-24T16:01:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25650K .......... .......... .......... .......... .......... 15% 25.5M 6s 2024-04-24T16:01:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25700K .......... .......... .......... .......... .......... 15% 29.0M 6s 2024-04-24T16:01:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25750K .......... .......... .......... .......... .......... 15% 35.8M 6s 2024-04-24T16:01:13.004 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25800K .......... .......... .......... .......... .......... 15% 29.3M 6s 2024-04-24T16:01:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25850K .......... .......... .......... .......... .......... 15% 27.4M 6s 2024-04-24T16:01:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25900K .......... .......... .......... .......... .......... 15% 24.4M 6s 2024-04-24T16:01:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 25950K .......... .......... .......... .......... .......... 15% 23.1M 6s 2024-04-24T16:01:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26000K .......... .......... .......... .......... .......... 15% 36.0M 6s 2024-04-24T16:01:13.013 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26050K .......... .......... .......... .......... .......... 15% 28.9M 6s 2024-04-24T16:01:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26100K .......... .......... .......... .......... .......... 15% 26.0M 6s 2024-04-24T16:01:13.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26150K .......... .......... .......... .......... .......... 15% 34.3M 6s 2024-04-24T16:01:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26200K .......... .......... .......... .......... .......... 15% 22.9M 6s 2024-04-24T16:01:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26250K .......... .......... .......... .......... .......... 15% 74.6M 6s 2024-04-24T16:01:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26300K .......... .......... .......... .......... .......... 15% 22.5M 6s 2024-04-24T16:01:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26350K .......... .......... .......... .......... .......... 15% 24.2M 6s 2024-04-24T16:01:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26400K .......... .......... .......... .......... .......... 15% 23.6M 6s 2024-04-24T16:01:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26450K .......... .......... .......... .......... .......... 15% 36.8M 6s 2024-04-24T16:01:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26500K .......... .......... .......... .......... .......... 15% 53.6M 6s 2024-04-24T16:01:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26550K .......... .......... .......... .......... .......... 15% 23.2M 6s 2024-04-24T16:01:13.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26600K .......... .......... .......... .......... .......... 15% 35.4M 6s 2024-04-24T16:01:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26650K .......... .......... .......... .......... .......... 15% 25.2M 6s 2024-04-24T16:01:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26700K .......... .......... .......... .......... .......... 15% 57.2M 6s 2024-04-24T16:01:13.037 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26750K .......... .......... .......... .......... .......... 15% 18.8M 6s 2024-04-24T16:01:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26800K .......... .......... .......... .......... .......... 15% 30.7M 6s 2024-04-24T16:01:13.040 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26850K .......... .......... .......... .......... .......... 15% 25.3M 6s 2024-04-24T16:01:13.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26900K .......... .......... .......... .......... .......... 15% 26.1M 6s 2024-04-24T16:01:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 26950K .......... .......... .......... .......... .......... 16% 40.3M 6s 2024-04-24T16:01:13.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27000K .......... .......... .......... .......... .......... 16% 34.8M 6s 2024-04-24T16:01:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27050K .......... .......... .......... .......... .......... 16% 29.1M 6s 2024-04-24T16:01:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27100K .......... .......... .......... .......... .......... 16% 29.4M 6s 2024-04-24T16:01:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27150K .......... .......... .......... .......... .......... 16% 17.8M 6s 2024-04-24T16:01:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27200K .......... .......... .......... .......... .......... 16% 44.8M 6s 2024-04-24T16:01:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27250K .......... .......... .......... .......... .......... 16% 35.8M 6s 2024-04-24T16:01:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27300K .......... .......... .......... .......... .......... 16% 29.0M 6s 2024-04-24T16:01:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27350K .......... .......... .......... .......... .......... 16% 30.0M 6s 2024-04-24T16:01:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27400K .......... .......... .......... .......... .......... 16% 31.9M 6s 2024-04-24T16:01:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27450K .......... .......... .......... .......... .......... 16% 23.0M 6s 2024-04-24T16:01:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27500K .......... .......... .......... .......... .......... 16% 36.6M 6s 2024-04-24T16:01:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27550K .......... .......... .......... .......... .......... 16% 15.7M 6s 2024-04-24T16:01:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27600K .......... .......... .......... .......... .......... 16% 90.6M 6s 2024-04-24T16:01:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27650K .......... .......... .......... .......... .......... 16% 26.6M 6s 2024-04-24T16:01:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27700K .......... .......... .......... .......... .......... 16% 28.0M 6s 2024-04-24T16:01:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27750K .......... .......... .......... .......... .......... 16% 25.4M 6s 2024-04-24T16:01:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27800K .......... .......... .......... .......... .......... 16% 56.6M 6s 2024-04-24T16:01:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27850K .......... .......... .......... .......... .......... 16% 29.0M 6s 2024-04-24T16:01:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27900K .......... .......... .......... .......... .......... 16% 30.0M 6s 2024-04-24T16:01:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 27950K .......... .......... .......... .......... .......... 16% 23.5M 5s 2024-04-24T16:01:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28000K .......... .......... .......... .......... .......... 16% 21.1M 5s 2024-04-24T16:01:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28050K .......... .......... .......... .......... .......... 16% 72.9M 5s 2024-04-24T16:01:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28100K .......... .......... .......... .......... .......... 16% 26.8M 5s 2024-04-24T16:01:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28150K .......... .......... .......... .......... .......... 16% 23.0M 5s 2024-04-24T16:01:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28200K .......... .......... .......... .......... .......... 16% 44.1M 5s 2024-04-24T16:01:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28250K .......... .......... .......... .......... .......... 16% 23.7M 5s 2024-04-24T16:01:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28300K .......... .......... .......... .......... .......... 16% 33.3M 5s 2024-04-24T16:01:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28350K .......... .......... .......... .......... .......... 16% 27.3M 5s 2024-04-24T16:01:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28400K .......... .......... .......... .......... .......... 16% 24.4M 5s 2024-04-24T16:01:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28450K .......... .......... .......... .......... .......... 16% 24.9M 5s 2024-04-24T16:01:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28500K .......... .......... .......... .......... .......... 16% 50.2M 5s 2024-04-24T16:01:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28550K .......... .......... .......... .......... .......... 16% 30.9M 5s 2024-04-24T16:01:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28600K .......... .......... .......... .......... .......... 16% 22.9M 5s 2024-04-24T16:01:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28650K .......... .......... .......... .......... .......... 17% 30.9M 5s 2024-04-24T16:01:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28700K .......... .......... .......... .......... .......... 17% 33.6M 5s 2024-04-24T16:01:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28750K .......... .......... .......... .......... .......... 17% 23.0M 5s 2024-04-24T16:01:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28800K .......... .......... .......... .......... .......... 17% 25.2M 5s 2024-04-24T16:01:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28850K .......... .......... .......... .......... .......... 17% 48.4M 5s 2024-04-24T16:01:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28900K .......... .......... .......... .......... .......... 17% 28.7M 5s 2024-04-24T16:01:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 28950K .......... .......... .......... .......... .......... 17% 34.1M 5s 2024-04-24T16:01:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29000K .......... .......... .......... .......... .......... 17% 30.6M 5s 2024-04-24T16:01:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29050K .......... .......... .......... .......... .......... 17% 23.2M 5s 2024-04-24T16:01:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29100K .......... .......... .......... .......... .......... 17% 31.7M 5s 2024-04-24T16:01:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29150K .......... .......... .......... .......... .......... 17% 21.3M 5s 2024-04-24T16:01:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29200K .......... .......... .......... .......... .......... 17% 31.6M 5s 2024-04-24T16:01:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29250K .......... .......... .......... .......... .......... 17% 24.6M 5s 2024-04-24T16:01:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29300K .......... .......... .......... .......... .......... 17% 53.6M 5s 2024-04-24T16:01:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29350K .......... .......... .......... .......... .......... 17% 32.8M 5s 2024-04-24T16:01:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29400K .......... .......... .......... .......... .......... 17% 24.7M 5s 2024-04-24T16:01:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29450K .......... .......... .......... .......... .......... 17% 25.2M 5s 2024-04-24T16:01:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29500K .......... .......... .......... .......... .......... 17% 33.7M 5s 2024-04-24T16:01:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29550K .......... .......... .......... .......... .......... 17% 25.2M 5s 2024-04-24T16:01:13.133 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29600K .......... .......... .......... .......... .......... 17% 22.6M 5s 2024-04-24T16:01:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29650K .......... .......... .......... .......... .......... 17% 23.7M 5s 2024-04-24T16:01:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29700K .......... .......... .......... .......... .......... 17% 27.3M 5s 2024-04-24T16:01:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29750K .......... .......... .......... .......... .......... 17% 105M 5s 2024-04-24T16:01:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29800K .......... .......... .......... .......... .......... 17% 22.6M 5s 2024-04-24T16:01:13.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29850K .......... .......... .......... .......... .......... 17% 39.5M 5s 2024-04-24T16:01:13.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29900K .......... .......... .......... .......... .......... 17% 24.3M 5s 2024-04-24T16:01:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 29950K .......... .......... .......... .......... .......... 17% 25.0M 5s 2024-04-24T16:01:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30000K .......... .......... .......... .......... .......... 17% 28.1M 5s 2024-04-24T16:01:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30050K .......... .......... .......... .......... .......... 17% 36.5M 5s 2024-04-24T16:01:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30100K .......... .......... .......... .......... .......... 17% 30.6M 5s 2024-04-24T16:01:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30150K .......... .......... .......... .......... .......... 17% 17.8M 5s 2024-04-24T16:01:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30200K .......... .......... .......... .......... .......... 17% 35.7M 5s 2024-04-24T16:01:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30250K .......... .......... .......... .......... .......... 17% 43.0M 5s 2024-04-24T16:01:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30300K .......... .......... .......... .......... .......... 18% 28.2M 5s 2024-04-24T16:01:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30350K .......... .......... .......... .......... .......... 18% 9.69M 5s 2024-04-24T16:01:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30400K .......... .......... .......... .......... .......... 18% 111M 5s 2024-04-24T16:01:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30450K .......... .......... .......... .......... .......... 18% 83.3M 5s 2024-04-24T16:01:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30500K .......... .......... .......... .......... .......... 18% 44.6M 5s 2024-04-24T16:01:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30550K .......... .......... .......... .......... .......... 18% 30.8M 5s 2024-04-24T16:01:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30600K .......... .......... .......... .......... .......... 18% 25.0M 5s 2024-04-24T16:01:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30650K .......... .......... .......... .......... .......... 18% 41.6M 5s 2024-04-24T16:01:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30700K .......... .......... .......... .......... .......... 18% 33.2M 5s 2024-04-24T16:01:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30750K .......... .......... .......... .......... .......... 18% 23.5M 5s 2024-04-24T16:01:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30800K .......... .......... .......... .......... .......... 18% 14.4M 5s 2024-04-24T16:01:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30850K .......... .......... .......... .......... .......... 18% 48.1M 5s 2024-04-24T16:01:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30900K .......... .......... .......... .......... .......... 18% 40.2M 5s 2024-04-24T16:01:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 30950K .......... .......... .......... .......... .......... 18% 39.4M 5s 2024-04-24T16:01:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31000K .......... .......... .......... .......... .......... 18% 28.8M 5s 2024-04-24T16:01:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31050K .......... .......... .......... .......... .......... 18% 27.9M 5s 2024-04-24T16:01:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31100K .......... .......... .......... .......... .......... 18% 19.1M 5s 2024-04-24T16:01:13.188 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31150K .......... .......... .......... .......... .......... 18% 22.7M 5s 2024-04-24T16:01:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31200K .......... .......... .......... .......... .......... 18% 30.4M 5s 2024-04-24T16:01:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31250K .......... .......... .......... .......... .......... 18% 91.6M 5s 2024-04-24T16:01:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31300K .......... .......... .......... .......... .......... 18% 35.7M 5s 2024-04-24T16:01:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31350K .......... .......... .......... .......... .......... 18% 21.7M 5s 2024-04-24T16:01:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31400K .......... .......... .......... .......... .......... 18% 18.9M 5s 2024-04-24T16:01:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31450K .......... .......... .......... .......... .......... 18% 50.4M 5s 2024-04-24T16:01:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31500K .......... .......... .......... .......... .......... 18% 42.7M 5s 2024-04-24T16:01:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31550K .......... .......... .......... .......... .......... 18% 24.6M 5s 2024-04-24T16:01:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31600K .......... .......... .......... .......... .......... 18% 34.9M 5s 2024-04-24T16:01:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31650K .......... .......... .......... .......... .......... 18% 24.9M 5s 2024-04-24T16:01:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31700K .......... .......... .......... .......... .......... 18% 30.1M 5s 2024-04-24T16:01:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31750K .......... .......... .......... .......... .......... 18% 22.7M 5s 2024-04-24T16:01:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31800K .......... .......... .......... .......... .......... 18% 44.1M 5s 2024-04-24T16:01:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31850K .......... .......... .......... .......... .......... 18% 27.8M 5s 2024-04-24T16:01:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31900K .......... .......... .......... .......... .......... 18% 33.5M 5s 2024-04-24T16:01:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 31950K .......... .......... .......... .......... .......... 18% 24.7M 5s 2024-04-24T16:01:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32000K .......... .......... .......... .......... .......... 19% 36.2M 5s 2024-04-24T16:01:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32050K .......... .......... .......... .......... .......... 19% 16.2M 5s 2024-04-24T16:01:13.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32100K .......... .......... .......... .......... .......... 19% 34.5M 5s 2024-04-24T16:01:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32150K .......... .......... .......... .......... .......... 19% 40.5M 5s 2024-04-24T16:01:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32200K .......... .......... .......... .......... .......... 19% 51.4M 5s 2024-04-24T16:01:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32250K .......... .......... .......... .......... .......... 19% 35.2M 5s 2024-04-24T16:01:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32300K .......... .......... .......... .......... .......... 19% 22.6M 5s 2024-04-24T16:01:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32350K .......... .......... .......... .......... .......... 19% 17.2M 5s 2024-04-24T16:01:13.228 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32400K .......... .......... .......... .......... .......... 19% 44.6M 5s 2024-04-24T16:01:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32450K .......... .......... .......... .......... .......... 19% 502K 6s 2024-04-24T16:01:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32500K .......... .......... .......... .......... .......... 19% 265M 6s 2024-04-24T16:01:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32550K .......... .......... .......... .......... .......... 19% 246M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32600K .......... .......... .......... .......... .......... 19% 260M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32650K .......... .......... .......... .......... .......... 19% 256M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32700K .......... .......... .......... .......... .......... 19% 250M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32750K .......... .......... .......... .......... .......... 19% 226M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32800K .......... .......... .......... .......... .......... 19% 268M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32850K .......... .......... .......... .......... .......... 19% 249M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32900K .......... .......... .......... .......... .......... 19% 290M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 32950K .......... .......... .......... .......... .......... 19% 254M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33000K .......... .......... .......... .......... .......... 19% 285M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33050K .......... .......... .......... .......... .......... 19% 237M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33100K .......... .......... .......... .......... .......... 19% 282M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33150K .......... .......... .......... .......... .......... 19% 221M 6s 2024-04-24T16:01:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33200K .......... .......... .......... .......... .......... 19% 283M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33250K .......... .......... .......... .......... .......... 19% 277M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33300K .......... .......... .......... .......... .......... 19% 272M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33350K .......... .......... .......... .......... .......... 19% 234M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33400K .......... .......... .......... .......... .......... 19% 296M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33450K .......... .......... .......... .......... .......... 19% 261M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33500K .......... .......... .......... .......... .......... 19% 281M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33550K .......... .......... .......... .......... .......... 19% 238M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33600K .......... .......... .......... .......... .......... 19% 279M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33650K .......... .......... .......... .......... .......... 19% 238M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33700K .......... .......... .......... .......... .......... 20% 287M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33750K .......... .......... .......... .......... .......... 20% 218M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33800K .......... .......... .......... .......... .......... 20% 279M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33850K .......... .......... .......... .......... .......... 20% 291M 5s 2024-04-24T16:01:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33900K .......... .......... .......... .......... .......... 20% 267M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 33950K .......... .......... .......... .......... .......... 20% 206M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34000K .......... .......... .......... .......... .......... 20% 260M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34050K .......... .......... .......... .......... .......... 20% 268M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34100K .......... .......... .......... .......... .......... 20% 282M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34150K .......... .......... .......... .......... .......... 20% 238M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34200K .......... .......... .......... .......... .......... 20% 252M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34250K .......... .......... .......... .......... .......... 20% 258M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34300K .......... .......... .......... .......... .......... 20% 266M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34350K .......... .......... .......... .......... .......... 20% 231M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34400K .......... .......... .......... .......... .......... 20% 281M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34450K .......... .......... .......... .......... .......... 20% 229M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34500K .......... .......... .......... .......... .......... 20% 287M 5s 2024-04-24T16:01:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34550K .......... .......... .......... .......... .......... 20% 217M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34600K .......... .......... .......... .......... .......... 20% 274M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34650K .......... .......... .......... .......... .......... 20% 276M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34700K .......... .......... .......... .......... .......... 20% 265M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34750K .......... .......... .......... .......... .......... 20% 233M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34800K .......... .......... .......... .......... .......... 20% 284M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34850K .......... .......... .......... .......... .......... 20% 276M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34900K .......... .......... .......... .......... .......... 20% 271M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 34950K .......... .......... .......... .......... .......... 20% 234M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35000K .......... .......... .......... .......... .......... 20% 248M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35050K .......... .......... .......... .......... .......... 20% 275M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35100K .......... .......... .......... .......... .......... 20% 294M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35150K .......... .......... .......... .......... .......... 20% 229M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35200K .......... .......... .......... .......... .......... 20% 264M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35250K .......... .......... .......... .......... .......... 20% 254M 5s 2024-04-24T16:01:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35300K .......... .......... .......... .......... .......... 20% 288M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35350K .......... .......... .......... .......... .......... 21% 246M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35400K .......... .......... .......... .......... .......... 21% 266M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35450K .......... .......... .......... .......... .......... 21% 1.94M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35500K .......... .......... .......... .......... .......... 21% 117M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35550K .......... .......... .......... .......... .......... 21% 9.14M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35600K .......... .......... .......... .......... .......... 21% 124M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35650K .......... .......... .......... .......... .......... 21% 110M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35700K .......... .......... .......... .......... .......... 21% 119M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35750K .......... .......... .......... .......... .......... 21% 117M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35800K .......... .......... .......... .......... .......... 21% 114M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35850K .......... .......... .......... .......... .......... 21% 41.3M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35900K .......... .......... .......... .......... .......... 21% 111M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 35950K .......... .......... .......... .......... .......... 21% 21.8M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36000K .......... .......... .......... .......... .......... 21% 32.9M 5s 2024-04-24T16:01:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36050K .......... .......... .......... .......... .......... 21% 43.8M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36100K .......... .......... .......... .......... .......... 21% 20.4M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36150K .......... .......... .......... .......... .......... 21% 58.4M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36200K .......... .......... .......... .......... .......... 21% 24.6M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36250K .......... .......... .......... .......... .......... 21% 42.9M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36300K .......... .......... .......... .......... .......... 21% 20.0M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36350K .......... .......... .......... .......... .......... 21% 30.1M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36400K .......... .......... .......... .......... .......... 21% 25.9M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36450K .......... .......... .......... .......... .......... 21% 37.3M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36500K .......... .......... .......... .......... .......... 21% 25.7M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36550K .......... .......... .......... .......... .......... 21% 38.5M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36600K .......... .......... .......... .......... .......... 21% 24.3M 5s 2024-04-24T16:01:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36650K .......... .......... .......... .......... .......... 21% 21.8M 5s 2024-04-24T16:01:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36700K .......... .......... .......... .......... .......... 21% 24.7M 5s 2024-04-24T16:01:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36750K .......... .......... .......... .......... .......... 21% 23.3M 5s 2024-04-24T16:01:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36800K .......... .......... .......... .......... .......... 21% 117M 5s 2024-04-24T16:01:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36850K .......... .......... .......... .......... .......... 21% 28.2M 5s 2024-04-24T16:01:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36900K .......... .......... .......... .......... .......... 21% 20.4M 5s 2024-04-24T16:01:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 36950K .......... .......... .......... .......... .......... 21% 26.6M 5s 2024-04-24T16:01:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37000K .......... .......... .......... .......... .......... 21% 22.7M 5s 2024-04-24T16:01:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37050K .......... .......... .......... .......... .......... 22% 98.1M 5s 2024-04-24T16:01:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37100K .......... .......... .......... .......... .......... 22% 37.8M 5s 2024-04-24T16:01:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37150K .......... .......... .......... .......... .......... 22% 26.4M 5s 2024-04-24T16:01:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37200K .......... .......... .......... .......... .......... 22% 16.4M 5s 2024-04-24T16:01:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37250K .......... .......... .......... .......... .......... 22% 22.8M 5s 2024-04-24T16:01:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37300K .......... .......... .......... .......... .......... 22% 116M 5s 2024-04-24T16:01:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37350K .......... .......... .......... .......... .......... 22% 24.6M 5s 2024-04-24T16:01:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37400K .......... .......... .......... .......... .......... 22% 25.5M 5s 2024-04-24T16:01:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37450K .......... .......... .......... .......... .......... 22% 24.3M 5s 2024-04-24T16:01:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37500K .......... .......... .......... .......... .......... 22% 25.3M 5s 2024-04-24T16:01:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37550K .......... .......... .......... .......... .......... 22% 23.8M 5s 2024-04-24T16:01:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37600K .......... .......... .......... .......... .......... 22% 106M 5s 2024-04-24T16:01:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37650K .......... .......... .......... .......... .......... 22% 21.8M 5s 2024-04-24T16:01:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37700K .......... .......... .......... .......... .......... 22% 39.3M 5s 2024-04-24T16:01:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37750K .......... .......... .......... .......... .......... 22% 20.1M 5s 2024-04-24T16:01:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37800K .......... .......... .......... .......... .......... 22% 102M 5s 2024-04-24T16:01:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37850K .......... .......... .......... .......... .......... 22% 20.6M 5s 2024-04-24T16:01:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37900K .......... .......... .......... .......... .......... 22% 29.4M 5s 2024-04-24T16:01:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 37950K .......... .......... .......... .......... .......... 22% 29.1M 5s 2024-04-24T16:01:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38000K .......... .......... .......... .......... .......... 22% 29.7M 5s 2024-04-24T16:01:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38050K .......... .......... .......... .......... .......... 22% 25.0M 5s 2024-04-24T16:01:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38100K .......... .......... .......... .......... .......... 22% 44.8M 5s 2024-04-24T16:01:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38150K .......... .......... .......... .......... .......... 22% 31.1M 5s 2024-04-24T16:01:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38200K .......... .......... .......... .......... .......... 22% 27.3M 5s 2024-04-24T16:01:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38250K .......... .......... .......... .......... .......... 22% 22.5M 5s 2024-04-24T16:01:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38300K .......... .......... .......... .......... .......... 22% 60.2M 5s 2024-04-24T16:01:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38350K .......... .......... .......... .......... .......... 22% 22.8M 5s 2024-04-24T16:01:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38400K .......... .......... .......... .......... .......... 22% 23.8M 5s 2024-04-24T16:01:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38450K .......... .......... .......... .......... .......... 22% 32.3M 5s 2024-04-24T16:01:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38500K .......... .......... .......... .......... .......... 22% 29.1M 5s 2024-04-24T16:01:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38550K .......... .......... .......... .......... .......... 22% 41.0M 5s 2024-04-24T16:01:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38600K .......... .......... .......... .......... .......... 22% 26.5M 5s 2024-04-24T16:01:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38650K .......... .......... .......... .......... .......... 22% 21.7M 5s 2024-04-24T16:01:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38700K .......... .......... .......... .......... .......... 22% 40.2M 5s 2024-04-24T16:01:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38750K .......... .......... .......... .......... .......... 23% 22.0M 5s 2024-04-24T16:01:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38800K .......... .......... .......... .......... .......... 23% 31.1M 5s 2024-04-24T16:01:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38850K .......... .......... .......... .......... .......... 23% 31.7M 5s 2024-04-24T16:01:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38900K .......... .......... .......... .......... .......... 23% 31.2M 5s 2024-04-24T16:01:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 38950K .......... .......... .......... .......... .......... 23% 28.5M 5s 2024-04-24T16:01:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39000K .......... .......... .......... .......... .......... 23% 25.1M 5s 2024-04-24T16:01:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39050K .......... .......... .......... .......... .......... 23% 41.9M 5s 2024-04-24T16:01:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39100K .......... .......... .......... .......... .......... 23% 31.4M 5s 2024-04-24T16:01:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39150K .......... .......... .......... .......... .......... 23% 20.2M 5s 2024-04-24T16:01:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39200K .......... .......... .......... .......... .......... 23% 36.1M 5s 2024-04-24T16:01:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39250K .......... .......... .......... .......... .......... 23% 24.8M 5s 2024-04-24T16:01:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39300K .......... .......... .......... .......... .......... 23% 21.9M 5s 2024-04-24T16:01:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39350K .......... .......... .......... .......... .......... 23% 56.0M 5s 2024-04-24T16:01:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39400K .......... .......... .......... .......... .......... 23% 33.0M 5s 2024-04-24T16:01:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39450K .......... .......... .......... .......... .......... 23% 32.4M 5s 2024-04-24T16:01:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39500K .......... .......... .......... .......... .......... 23% 22.3M 5s 2024-04-24T16:01:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39550K .......... .......... .......... .......... .......... 23% 28.3M 5s 2024-04-24T16:01:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39600K .......... .......... .......... .......... .......... 23% 37.8M 5s 2024-04-24T16:01:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39650K .......... .......... .......... .......... .......... 23% 26.6M 5s 2024-04-24T16:01:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39700K .......... .......... .......... .......... .......... 23% 26.9M 5s 2024-04-24T16:01:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39750K .......... .......... .......... .......... .......... 23% 28.4M 5s 2024-04-24T16:01:13.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39800K .......... .......... .......... .......... .......... 23% 48.8M 5s 2024-04-24T16:01:13.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39850K .......... .......... .......... .......... .......... 23% 28.0M 5s 2024-04-24T16:01:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39900K .......... .......... .......... .......... .......... 23% 28.0M 5s 2024-04-24T16:01:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 39950K .......... .......... .......... .......... .......... 23% 20.1M 5s 2024-04-24T16:01:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40000K .......... .......... .......... .......... .......... 23% 27.7M 5s 2024-04-24T16:01:13.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40050K .......... .......... .......... .......... .......... 23% 54.2M 5s 2024-04-24T16:01:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40100K .......... .......... .......... .......... .......... 23% 23.9M 5s 2024-04-24T16:01:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40150K .......... .......... .......... .......... .......... 23% 37.0M 5s 2024-04-24T16:01:13.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40200K .......... .......... .......... .......... .......... 23% 24.3M 5s 2024-04-24T16:01:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40250K .......... .......... .......... .......... .......... 23% 28.5M 5s 2024-04-24T16:01:13.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40300K .......... .......... .......... .......... .......... 23% 48.3M 5s 2024-04-24T16:01:13.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40350K .......... .......... .......... .......... .......... 23% 19.1M 5s 2024-04-24T16:01:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40400K .......... .......... .......... .......... .......... 23% 32.3M 5s 2024-04-24T16:01:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40450K .......... .......... .......... .......... .......... 24% 28.9M 5s 2024-04-24T16:01:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40500K .......... .......... .......... .......... .......... 24% 31.8M 5s 2024-04-24T16:01:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40550K .......... .......... .......... .......... .......... 24% 33.0M 5s 2024-04-24T16:01:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40600K .......... .......... .......... .......... .......... 24% 28.8M 5s 2024-04-24T16:01:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40650K .......... .......... .......... .......... .......... 24% 27.5M 5s 2024-04-24T16:01:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40700K .......... .......... .......... .......... .......... 24% 25.2M 5s 2024-04-24T16:01:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40750K .......... .......... .......... .......... .......... 24% 21.2M 5s 2024-04-24T16:01:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40800K .......... .......... .......... .......... .......... 24% 42.0M 5s 2024-04-24T16:01:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40850K .......... .......... .......... .......... .......... 24% 37.7M 5s 2024-04-24T16:01:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40900K .......... .......... .......... .......... .......... 24% 24.6M 5s 2024-04-24T16:01:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 40950K .......... .......... .......... .......... .......... 24% 23.4M 5s 2024-04-24T16:01:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41000K .......... .......... .......... .......... .......... 24% 46.8M 5s 2024-04-24T16:01:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41050K .......... .......... .......... .......... .......... 24% 21.1M 5s 2024-04-24T16:01:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41100K .......... .......... .......... .......... .......... 24% 36.4M 5s 2024-04-24T16:01:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41150K .......... .......... .......... .......... .......... 24% 28.0M 5s 2024-04-24T16:01:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41200K .......... .......... .......... .......... .......... 24% 21.7M 5s 2024-04-24T16:01:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41250K .......... .......... .......... .......... .......... 24% 54.8M 5s 2024-04-24T16:01:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41300K .......... .......... .......... .......... .......... 24% 21.7M 5s 2024-04-24T16:01:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41350K .......... .......... .......... .......... .......... 24% 44.6M 5s 2024-04-24T16:01:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41400K .......... .......... .......... .......... .......... 24% 24.6M 5s 2024-04-24T16:01:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41450K .......... .......... .......... .......... .......... 24% 25.5M 5s 2024-04-24T16:01:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41500K .......... .......... .......... .......... .......... 24% 44.3M 5s 2024-04-24T16:01:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41550K .......... .......... .......... .......... .......... 24% 19.6M 5s 2024-04-24T16:01:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41600K .......... .......... .......... .......... .......... 24% 25.2M 5s 2024-04-24T16:01:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41650K .......... .......... .......... .......... .......... 24% 56.1M 5s 2024-04-24T16:01:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41700K .......... .......... .......... .......... .......... 24% 24.3M 5s 2024-04-24T16:01:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41750K .......... .......... .......... .......... .......... 24% 37.7M 5s 2024-04-24T16:01:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41800K .......... .......... .......... .......... .......... 24% 23.5M 5s 2024-04-24T16:01:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41850K .......... .......... .......... .......... .......... 24% 43.1M 5s 2024-04-24T16:01:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41900K .......... .......... .......... .......... .......... 24% 30.0M 5s 2024-04-24T16:01:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 41950K .......... .......... .......... .......... .......... 24% 21.7M 5s 2024-04-24T16:01:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42000K .......... .......... .......... .......... .......... 24% 24.1M 5s 2024-04-24T16:01:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42050K .......... .......... .......... .......... .......... 24% 32.9M 5s 2024-04-24T16:01:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42100K .......... .......... .......... .......... .......... 25% 28.5M 5s 2024-04-24T16:01:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42150K .......... .......... .......... .......... .......... 25% 50.7M 5s 2024-04-24T16:01:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42200K .......... .......... .......... .......... .......... 25% 21.2M 5s 2024-04-24T16:01:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42250K .......... .......... .......... .......... .......... 25% 42.7M 5s 2024-04-24T16:01:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42300K .......... .......... .......... .......... .......... 25% 24.3M 5s 2024-04-24T16:01:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42350K .......... .......... .......... .......... .......... 25% 25.5M 5s 2024-04-24T16:01:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42400K .......... .......... .......... .......... .......... 25% 46.2M 5s 2024-04-24T16:01:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42450K .......... .......... .......... .......... .......... 25% 23.0M 5s 2024-04-24T16:01:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42500K .......... .......... .......... .......... .......... 25% 36.5M 5s 2024-04-24T16:01:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42550K .......... .......... .......... .......... .......... 25% 24.1M 5s 2024-04-24T16:01:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42600K .......... .......... .......... .......... .......... 25% 24.0M 5s 2024-04-24T16:01:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42650K .......... .......... .......... .......... .......... 25% 65.4M 5s 2024-04-24T16:01:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42700K .......... .......... .......... .......... .......... 25% 29.7M 5s 2024-04-24T16:01:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42750K .......... .......... .......... .......... .......... 25% 18.9M 5s 2024-04-24T16:01:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42800K .......... .......... .......... .......... .......... 25% 35.6M 5s 2024-04-24T16:01:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42850K .......... .......... .......... .......... .......... 25% 22.2M 5s 2024-04-24T16:01:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42900K .......... .......... .......... .......... .......... 25% 62.6M 5s 2024-04-24T16:01:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 42950K .......... .......... .......... .......... .......... 25% 28.5M 5s 2024-04-24T16:01:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43000K .......... .......... .......... .......... .......... 25% 29.7M 5s 2024-04-24T16:01:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43050K .......... .......... .......... .......... .......... 25% 22.5M 5s 2024-04-24T16:01:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43100K .......... .......... .......... .......... .......... 25% 56.7M 5s 2024-04-24T16:01:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43150K .......... .......... .......... .......... .......... 25% 15.2M 5s 2024-04-24T16:01:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43200K .......... .......... .......... .......... .......... 25% 48.2M 5s 2024-04-24T16:01:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43250K .......... .......... .......... .......... .......... 25% 24.6M 5s 2024-04-24T16:01:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43300K .......... .......... .......... .......... .......... 25% 34.3M 5s 2024-04-24T16:01:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43350K .......... .......... .......... .......... .......... 25% 20.6M 5s 2024-04-24T16:01:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43400K .......... .......... .......... .......... .......... 25% 89.3M 5s 2024-04-24T16:01:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43450K .......... .......... .......... .......... .......... 25% 25.0M 5s 2024-04-24T16:01:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43500K .......... .......... .......... .......... .......... 25% 40.5M 5s 2024-04-24T16:01:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43550K .......... .......... .......... .......... .......... 25% 22.0M 5s 2024-04-24T16:01:13.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43600K .......... .......... .......... .......... .......... 25% 23.1M 5s 2024-04-24T16:01:13.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43650K .......... .......... .......... .......... .......... 25% 49.3M 5s 2024-04-24T16:01:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43700K .......... .......... .......... .......... .......... 25% 23.6M 5s 2024-04-24T16:01:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43750K .......... .......... .......... .......... .......... 25% 33.5M 5s 2024-04-24T16:01:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43800K .......... .......... .......... .......... .......... 26% 20.4M 5s 2024-04-24T16:01:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43850K .......... .......... .......... .......... .......... 26% 26.8M 5s 2024-04-24T16:01:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43900K .......... .......... .......... .......... .......... 26% 67.9M 5s 2024-04-24T16:01:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 43950K .......... .......... .......... .......... .......... 26% 22.9M 5s 2024-04-24T16:01:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44000K .......... .......... .......... .......... .......... 26% 28.8M 5s 2024-04-24T16:01:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44050K .......... .......... .......... .......... .......... 26% 29.4M 5s 2024-04-24T16:01:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44100K .......... .......... .......... .......... .......... 26% 24.5M 5s 2024-04-24T16:01:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44150K .......... .......... .......... .......... .......... 26% 51.7M 5s 2024-04-24T16:01:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44200K .......... .......... .......... .......... .......... 26% 26.3M 5s 2024-04-24T16:01:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44250K .......... .......... .......... .......... .......... 26% 30.6M 5s 2024-04-24T16:01:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44300K .......... .......... .......... .......... .......... 26% 25.0M 5s 2024-04-24T16:01:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44350K .......... .......... .......... .......... .......... 26% 23.0M 5s 2024-04-24T16:01:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44400K .......... .......... .......... .......... .......... 26% 24.2M 5s 2024-04-24T16:01:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44450K .......... .......... .......... .......... .......... 26% 42.5M 5s 2024-04-24T16:01:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44500K .......... .......... .......... .......... .......... 26% 12.4M 5s 2024-04-24T16:01:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44550K .......... .......... .......... .......... .......... 26% 37.6M 5s 2024-04-24T16:01:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44600K .......... .......... .......... .......... .......... 26% 43.2M 5s 2024-04-24T16:01:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44650K .......... .......... .......... .......... .......... 26% 45.0M 5s 2024-04-24T16:01:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44700K .......... .......... .......... .......... .......... 26% 54.9M 5s 2024-04-24T16:01:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44750K .......... .......... .......... .......... .......... 26% 32.0M 5s 2024-04-24T16:01:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44800K .......... .......... .......... .......... .......... 26% 28.8M 5s 2024-04-24T16:01:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44850K .......... .......... .......... .......... .......... 26% 16.6M 5s 2024-04-24T16:01:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44900K .......... .......... .......... .......... .......... 26% 79.0M 5s 2024-04-24T16:01:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 44950K .......... .......... .......... .......... .......... 26% 27.2M 5s 2024-04-24T16:01:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45000K .......... .......... .......... .......... .......... 26% 27.1M 5s 2024-04-24T16:01:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45050K .......... .......... .......... .......... .......... 26% 31.7M 5s 2024-04-24T16:01:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45100K .......... .......... .......... .......... .......... 26% 25.8M 5s 2024-04-24T16:01:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45150K .......... .......... .......... .......... .......... 26% 18.5M 5s 2024-04-24T16:01:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45200K .......... .......... .......... .......... .......... 26% 44.2M 5s 2024-04-24T16:01:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45250K .......... .......... .......... .......... .......... 26% 33.2M 5s 2024-04-24T16:01:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45300K .......... .......... .......... .......... .......... 26% 36.7M 5s 2024-04-24T16:01:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45350K .......... .......... .......... .......... .......... 26% 23.9M 5s 2024-04-24T16:01:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45400K .......... .......... .......... .......... .......... 26% 24.1M 5s 2024-04-24T16:01:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45450K .......... .......... .......... .......... .......... 26% 60.7M 5s 2024-04-24T16:01:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45500K .......... .......... .......... .......... .......... 27% 32.7M 5s 2024-04-24T16:01:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45550K .......... .......... .......... .......... .......... 27% 17.5M 5s 2024-04-24T16:01:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45600K .......... .......... .......... .......... .......... 27% 28.0M 5s 2024-04-24T16:01:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45650K .......... .......... .......... .......... .......... 27% 29.1M 5s 2024-04-24T16:01:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45700K .......... .......... .......... .......... .......... 27% 49.4M 5s 2024-04-24T16:01:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45750K .......... .......... .......... .......... .......... 27% 24.7M 5s 2024-04-24T16:01:13.709 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45800K .......... .......... .......... .......... .......... 27% 25.5M 5s 2024-04-24T16:01:13.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45850K .......... .......... .......... .......... .......... 27% 30.8M 5s 2024-04-24T16:01:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45900K .......... .......... .......... .......... .......... 27% 45.5M 5s 2024-04-24T16:01:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 45950K .......... .......... .......... .......... .......... 27% 22.1M 5s 2024-04-24T16:01:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46000K .......... .......... .......... .......... .......... 27% 27.3M 5s 2024-04-24T16:01:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46050K .......... .......... .......... .......... .......... 27% 38.8M 5s 2024-04-24T16:01:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46100K .......... .......... .......... .......... .......... 27% 15.2M 5s 2024-04-24T16:01:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46150K .......... .......... .......... .......... .......... 27% 28.2M 5s 2024-04-24T16:01:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46200K .......... .......... .......... .......... .......... 27% 32.1M 5s 2024-04-24T16:01:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46250K .......... .......... .......... .......... .......... 27% 41.4M 5s 2024-04-24T16:01:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46300K .......... .......... .......... .......... .......... 27% 42.7M 5s 2024-04-24T16:01:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46350K .......... .......... .......... .......... .......... 27% 21.9M 5s 2024-04-24T16:01:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46400K .......... .......... .......... .......... .......... 27% 21.5M 5s 2024-04-24T16:01:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46450K .......... .......... .......... .......... .......... 27% 45.4M 5s 2024-04-24T16:01:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46500K .......... .......... .......... .......... .......... 27% 42.6M 5s 2024-04-24T16:01:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46550K .......... .......... .......... .......... .......... 27% 22.8M 5s 2024-04-24T16:01:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46600K .......... .......... .......... .......... .......... 27% 26.8M 5s 2024-04-24T16:01:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46650K .......... .......... .......... .......... .......... 27% 56.5M 5s 2024-04-24T16:01:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46700K .......... .......... .......... .......... .......... 27% 35.0M 5s 2024-04-24T16:01:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46750K .......... .......... .......... .......... .......... 27% 17.2M 5s 2024-04-24T16:01:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46800K .......... .......... .......... .......... .......... 27% 32.6M 5s 2024-04-24T16:01:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46850K .......... .......... .......... .......... .......... 27% 27.3M 5s 2024-04-24T16:01:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46900K .......... .......... .......... .......... .......... 27% 40.3M 5s 2024-04-24T16:01:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 46950K .......... .......... .......... .......... .......... 27% 35.0M 5s 2024-04-24T16:01:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47000K .......... .......... .......... .......... .......... 27% 30.1M 5s 2024-04-24T16:01:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47050K .......... .......... .......... .......... .......... 27% 23.9M 5s 2024-04-24T16:01:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47100K .......... .......... .......... .......... .......... 27% 28.3M 5s 2024-04-24T16:01:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47150K .......... .......... .......... .......... .......... 28% 30.9M 5s 2024-04-24T16:01:13.757 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47200K .......... .......... .......... .......... .......... 28% 21.2M 5s 2024-04-24T16:01:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47250K .......... .......... .......... .......... .......... 28% 42.0M 5s 2024-04-24T16:01:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47300K .......... .......... .......... .......... .......... 28% 32.6M 5s 2024-04-24T16:01:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47350K .......... .......... .......... .......... .......... 28% 30.0M 5s 2024-04-24T16:01:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47400K .......... .......... .......... .......... .......... 28% 20.9M 5s 2024-04-24T16:01:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47450K .......... .......... .......... .......... .......... 28% 69.8M 5s 2024-04-24T16:01:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47500K .......... .......... .......... .......... .......... 28% 23.9M 5s 2024-04-24T16:01:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47550K .......... .......... .......... .......... .......... 28% 27.4M 5s 2024-04-24T16:01:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47600K .......... .......... .......... .......... .......... 28% 21.3M 5s 2024-04-24T16:01:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47650K .......... .......... .......... .......... .......... 28% 39.4M 5s 2024-04-24T16:01:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47700K .......... .......... .......... .......... .......... 28% 26.6M 5s 2024-04-24T16:01:13.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47750K .......... .......... .......... .......... .......... 28% 42.1M 5s 2024-04-24T16:01:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47800K .......... .......... .......... .......... .......... 28% 19.4M 5s 2024-04-24T16:01:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47850K .......... .......... .......... .......... .......... 28% 35.0M 5s 2024-04-24T16:01:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47900K .......... .......... .......... .......... .......... 28% 42.3M 5s 2024-04-24T16:01:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 47950K .......... .......... .......... .......... .......... 28% 20.2M 5s 2024-04-24T16:01:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48000K .......... .......... .......... .......... .......... 28% 41.6M 5s 2024-04-24T16:01:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48050K .......... .......... .......... .......... .......... 28% 27.1M 5s 2024-04-24T16:01:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48100K .......... .......... .......... .......... .......... 28% 35.5M 5s 2024-04-24T16:01:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48150K .......... .......... .......... .......... .......... 28% 24.6M 5s 2024-04-24T16:01:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48200K .......... .......... .......... .......... .......... 28% 49.2M 5s 2024-04-24T16:01:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48250K .......... .......... .......... .......... .......... 28% 24.3M 5s 2024-04-24T16:01:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48300K .......... .......... .......... .......... .......... 28% 37.6M 5s 2024-04-24T16:01:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48350K .......... .......... .......... .......... .......... 28% 22.7M 5s 2024-04-24T16:01:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48400K .......... .......... .......... .......... .......... 28% 26.5M 5s 2024-04-24T16:01:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48450K .......... .......... .......... .......... .......... 28% 24.2M 4s 2024-04-24T16:01:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48500K .......... .......... .......... .......... .......... 28% 46.2M 4s 2024-04-24T16:01:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48550K .......... .......... .......... .......... .......... 28% 25.4M 4s 2024-04-24T16:01:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48600K .......... .......... .......... .......... .......... 28% 34.7M 4s 2024-04-24T16:01:13.805 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48650K .......... .......... .......... .......... .......... 28% 22.0M 4s 2024-04-24T16:01:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48700K .......... .......... .......... .......... .......... 28% 63.6M 4s 2024-04-24T16:01:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48750K .......... .......... .......... .......... .......... 28% 23.1M 4s 2024-04-24T16:01:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48800K .......... .......... .......... .......... .......... 28% 25.2M 4s 2024-04-24T16:01:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48850K .......... .......... .......... .......... .......... 29% 23.9M 4s 2024-04-24T16:01:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48900K .......... .......... .......... .......... .......... 29% 38.3M 4s 2024-04-24T16:01:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 48950K .......... .......... .......... .......... .......... 29% 28.2M 4s 2024-04-24T16:01:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49000K .......... .......... .......... .......... .......... 29% 39.1M 4s 2024-04-24T16:01:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49050K .......... .......... .......... .......... .......... 29% 23.5M 4s 2024-04-24T16:01:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49100K .......... .......... .......... .......... .......... 29% 37.7M 4s 2024-04-24T16:01:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49150K .......... .......... .......... .......... .......... 29% 24.5M 4s 2024-04-24T16:01:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49200K .......... .......... .......... .......... .......... 29% 23.5M 4s 2024-04-24T16:01:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49250K .......... .......... .......... .......... .......... 29% 45.4M 4s 2024-04-24T16:01:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49300K .......... .......... .......... .......... .......... 29% 25.5M 4s 2024-04-24T16:01:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49350K .......... .......... .......... .......... .......... 29% 29.1M 4s 2024-04-24T16:01:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49400K .......... .......... .......... .......... .......... 29% 29.3M 4s 2024-04-24T16:01:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49450K .......... .......... .......... .......... .......... 29% 24.2M 4s 2024-04-24T16:01:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49500K .......... .......... .......... .......... .......... 29% 34.7M 4s 2024-04-24T16:01:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49550K .......... .......... .......... .......... .......... 29% 29.4M 4s 2024-04-24T16:01:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49600K .......... .......... .......... .......... .......... 29% 24.7M 4s 2024-04-24T16:01:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49650K .......... .......... .......... .......... .......... 29% 33.4M 4s 2024-04-24T16:01:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49700K .......... .......... .......... .......... .......... 29% 24.1M 4s 2024-04-24T16:01:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49750K .......... .......... .......... .......... .......... 29% 58.1M 4s 2024-04-24T16:01:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49800K .......... .......... .......... .......... .......... 29% 23.8M 4s 2024-04-24T16:01:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49850K .......... .......... .......... .......... .......... 29% 36.7M 4s 2024-04-24T16:01:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49900K .......... .......... .......... .......... .......... 29% 24.1M 4s 2024-04-24T16:01:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 49950K .......... .......... .......... .......... .......... 29% 22.1M 4s 2024-04-24T16:01:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50000K .......... .......... .......... .......... .......... 29% 23.8M 4s 2024-04-24T16:01:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50050K .......... .......... .......... .......... .......... 29% 71.5M 4s 2024-04-24T16:01:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50100K .......... .......... .......... .......... .......... 29% 22.3M 4s 2024-04-24T16:01:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50150K .......... .......... .......... .......... .......... 29% 38.5M 4s 2024-04-24T16:01:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50200K .......... .......... .......... .......... .......... 29% 25.3M 4s 2024-04-24T16:01:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50250K .......... .......... .......... .......... .......... 29% 44.2M 4s 2024-04-24T16:01:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50300K .......... .......... .......... .......... .......... 29% 27.5M 4s 2024-04-24T16:01:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50350K .......... .......... .......... .......... .......... 29% 22.5M 4s 2024-04-24T16:01:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50400K .......... .......... .......... .......... .......... 29% 28.6M 4s 2024-04-24T16:01:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50450K .......... .......... .......... .......... .......... 29% 24.1M 4s 2024-04-24T16:01:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50500K .......... .......... .......... .......... .......... 29% 29.9M 4s 2024-04-24T16:01:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50550K .......... .......... .......... .......... .......... 30% 40.4M 4s 2024-04-24T16:01:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50600K .......... .......... .......... .......... .......... 30% 34.6M 4s 2024-04-24T16:01:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50650K .......... .......... .......... .......... .......... 30% 22.9M 4s 2024-04-24T16:01:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50700K .......... .......... .......... .......... .......... 30% 71.0M 4s 2024-04-24T16:01:13.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50750K .......... .......... .......... .......... .......... 30% 16.4M 4s 2024-04-24T16:01:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50800K .......... .......... .......... .......... .......... 30% 46.4M 4s 2024-04-24T16:01:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50850K .......... .......... .......... .......... .......... 30% 34.1M 4s 2024-04-24T16:01:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50900K .......... .......... .......... .......... .......... 30% 21.6M 4s 2024-04-24T16:01:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 50950K .......... .......... .......... .......... .......... 30% 33.6M 4s 2024-04-24T16:01:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51000K .......... .......... .......... .......... .......... 30% 39.7M 4s 2024-04-24T16:01:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51050K .......... .......... .......... .......... .......... 30% 26.0M 4s 2024-04-24T16:01:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51100K .......... .......... .......... .......... .......... 30% 34.8M 4s 2024-04-24T16:01:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51150K .......... .......... .......... .......... .......... 30% 21.8M 4s 2024-04-24T16:01:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51200K .......... .......... .......... .......... .......... 30% 26.3M 4s 2024-04-24T16:01:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51250K .......... .......... .......... .......... .......... 30% 25.0M 4s 2024-04-24T16:01:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51300K .......... .......... .......... .......... .......... 30% 54.0M 4s 2024-04-24T16:01:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51350K .......... .......... .......... .......... .......... 30% 23.3M 4s 2024-04-24T16:01:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51400K .......... .......... .......... .......... .......... 30% 34.4M 4s 2024-04-24T16:01:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51450K .......... .......... .......... .......... .......... 30% 23.6M 4s 2024-04-24T16:01:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51500K .......... .......... .......... .......... .......... 30% 57.1M 4s 2024-04-24T16:01:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51550K .......... .......... .......... .......... .......... 30% 21.9M 4s 2024-04-24T16:01:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51600K .......... .......... .......... .......... .......... 30% 28.3M 4s 2024-04-24T16:01:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51650K .......... .......... .......... .......... .......... 30% 32.6M 4s 2024-04-24T16:01:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51700K .......... .......... .......... .......... .......... 30% 23.8M 4s 2024-04-24T16:01:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51750K .......... .......... .......... .......... .......... 30% 25.8M 4s 2024-04-24T16:01:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51800K .......... .......... .......... .......... .......... 30% 42.7M 4s 2024-04-24T16:01:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51850K .......... .......... .......... .......... .......... 30% 24.2M 4s 2024-04-24T16:01:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51900K .......... .......... .......... .......... .......... 30% 30.6M 4s 2024-04-24T16:01:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 51950K .......... .......... .......... .......... .......... 30% 22.6M 4s 2024-04-24T16:01:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52000K .......... .......... .......... .......... .......... 30% 31.4M 4s 2024-04-24T16:01:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52050K .......... .......... .......... .......... .......... 30% 46.5M 4s 2024-04-24T16:01:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52100K .......... .......... .......... .......... .......... 30% 22.1M 4s 2024-04-24T16:01:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52150K .......... .......... .......... .......... .......... 30% 41.1M 4s 2024-04-24T16:01:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52200K .......... .......... .......... .......... .......... 30% 22.9M 4s 2024-04-24T16:01:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52250K .......... .......... .......... .......... .......... 31% 23.9M 4s 2024-04-24T16:01:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52300K .......... .......... .......... .......... .......... 31% 75.4M 4s 2024-04-24T16:01:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52350K .......... .......... .......... .......... .......... 31% 24.6M 4s 2024-04-24T16:01:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52400K .......... .......... .......... .......... .......... 31% 18.8M 4s 2024-04-24T16:01:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52450K .......... .......... .......... .......... .......... 31% 36.0M 4s 2024-04-24T16:01:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52500K .......... .......... .......... .......... .......... 31% 31.0M 4s 2024-04-24T16:01:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52550K .......... .......... .......... .......... .......... 31% 53.7M 4s 2024-04-24T16:01:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52600K .......... .......... .......... .......... .......... 31% 25.4M 4s 2024-04-24T16:01:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52650K .......... .......... .......... .......... .......... 31% 34.4M 4s 2024-04-24T16:01:13.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52700K .......... .......... .......... .......... .......... 31% 25.1M 4s 2024-04-24T16:01:13.943 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52750K .......... .......... .......... .......... .......... 31% 23.4M 4s 2024-04-24T16:01:13.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52800K .......... .......... .......... .......... .......... 31% 20.0M 4s 2024-04-24T16:01:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52850K .......... .......... .......... .......... .......... 31% 46.9M 4s 2024-04-24T16:01:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52900K .......... .......... .......... .......... .......... 31% 31.2M 4s 2024-04-24T16:01:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 52950K .......... .......... .......... .......... .......... 31% 29.5M 4s 2024-04-24T16:01:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53000K .......... .......... .......... .......... .......... 31% 25.7M 4s 2024-04-24T16:01:13.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53050K .......... .......... .......... .......... .......... 31% 52.8M 4s 2024-04-24T16:01:13.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53100K .......... .......... .......... .......... .......... 31% 23.6M 4s 2024-04-24T16:01:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53150K .......... .......... .......... .......... .......... 31% 22.8M 4s 2024-04-24T16:01:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53200K .......... .......... .......... .......... .......... 31% 24.6M 4s 2024-04-24T16:01:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53250K .......... .......... .......... .......... .......... 31% 34.2M 4s 2024-04-24T16:01:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53300K .......... .......... .......... .......... .......... 31% 46.8M 4s 2024-04-24T16:01:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53350K .......... .......... .......... .......... .......... 31% 28.3M 4s 2024-04-24T16:01:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53400K .......... .......... .......... .......... .......... 31% 22.4M 4s 2024-04-24T16:01:13.967 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53450K .......... .......... .......... .......... .......... 31% 27.0M 4s 2024-04-24T16:01:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53500K .......... .......... .......... .......... .......... 31% 32.5M 4s 2024-04-24T16:01:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53550K .......... .......... .......... .......... .......... 31% 26.6M 4s 2024-04-24T16:01:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53600K .......... .......... .......... .......... .......... 31% 32.7M 4s 2024-04-24T16:01:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53650K .......... .......... .......... .......... .......... 31% 32.4M 4s 2024-04-24T16:01:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53700K .......... .......... .......... .......... .......... 31% 21.0M 4s 2024-04-24T16:01:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53750K .......... .......... .......... .......... .......... 31% 22.4M 4s 2024-04-24T16:01:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53800K .......... .......... .......... .......... .......... 31% 114M 4s 2024-04-24T16:01:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53850K .......... .......... .......... .......... .......... 31% 27.6M 4s 2024-04-24T16:01:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53900K .......... .......... .......... .......... .......... 32% 21.8M 4s 2024-04-24T16:01:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 53950K .......... .......... .......... .......... .......... 32% 37.4M 4s 2024-04-24T16:01:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54000K .......... .......... .......... .......... .......... 32% 20.6M 4s 2024-04-24T16:01:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54050K .......... .......... .......... .......... .......... 32% 41.0M 4s 2024-04-24T16:01:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54100K .......... .......... .......... .......... .......... 32% 36.0M 4s 2024-04-24T16:01:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54150K .......... .......... .......... .......... .......... 32% 28.6M 4s 2024-04-24T16:01:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54200K .......... .......... .......... .......... .......... 32% 32.4M 4s 2024-04-24T16:01:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54250K .......... .......... .......... .......... .......... 32% 20.6M 4s 2024-04-24T16:01:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54300K .......... .......... .......... .......... .......... 32% 66.5M 4s 2024-04-24T16:01:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54350K .......... .......... .......... .......... .......... 32% 22.1M 4s 2024-04-24T16:01:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54400K .......... .......... .......... .......... .......... 32% 23.0M 4s 2024-04-24T16:01:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54450K .......... .......... .......... .......... .......... 32% 29.0M 4s 2024-04-24T16:01:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54500K .......... .......... .......... .......... .......... 32% 28.2M 4s 2024-04-24T16:01:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54550K .......... .......... .......... .......... .......... 32% 53.3M 4s 2024-04-24T16:01:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54600K .......... .......... .......... .......... .......... 32% 28.7M 4s 2024-04-24T16:01:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54650K .......... .......... .......... .......... .......... 32% 27.1M 4s 2024-04-24T16:01:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54700K .......... .......... .......... .......... .......... 32% 23.9M 4s 2024-04-24T16:01:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54750K .......... .......... .......... .......... .......... 32% 21.7M 4s 2024-04-24T16:01:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54800K .......... .......... .......... .......... .......... 32% 35.7M 4s 2024-04-24T16:01:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54850K .......... .......... .......... .......... .......... 32% 35.7M 4s 2024-04-24T16:01:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54900K .......... .......... .......... .......... .......... 32% 29.5M 4s 2024-04-24T16:01:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 54950K .......... .......... .......... .......... .......... 32% 25.0M 4s 2024-04-24T16:01:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55000K .......... .......... .......... .......... .......... 32% 40.2M 4s 2024-04-24T16:01:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55050K .......... .......... .......... .......... .......... 32% 20.5M 4s 2024-04-24T16:01:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55100K .......... .......... .......... .......... .......... 32% 58.0M 4s 2024-04-24T16:01:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55150K .......... .......... .......... .......... .......... 32% 19.8M 4s 2024-04-24T16:01:14.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55200K .......... .......... .......... .......... .......... 32% 27.6M 4s 2024-04-24T16:01:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55250K .......... .......... .......... .......... .......... 32% 37.7M 4s 2024-04-24T16:01:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55300K .......... .......... .......... .......... .......... 32% 28.9M 4s 2024-04-24T16:01:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55350K .......... .......... .......... .......... .......... 32% 38.7M 4s 2024-04-24T16:01:14.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55400K .......... .......... .......... .......... .......... 32% 23.6M 4s 2024-04-24T16:01:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55450K .......... .......... .......... .......... .......... 32% 35.9M 4s 2024-04-24T16:01:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55500K .......... .......... .......... .......... .......... 32% 26.6M 4s 2024-04-24T16:01:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55550K .......... .......... .......... .......... .......... 32% 22.5M 4s 2024-04-24T16:01:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55600K .......... .......... .......... .......... .......... 33% 25.4M 4s 2024-04-24T16:01:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55650K .......... .......... .......... .......... .......... 33% 42.9M 4s 2024-04-24T16:01:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55700K .......... .......... .......... .......... .......... 33% 23.1M 4s 2024-04-24T16:01:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55750K .......... .......... .......... .......... .......... 33% 53.9M 4s 2024-04-24T16:01:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55800K .......... .......... .......... .......... .......... 33% 21.8M 4s 2024-04-24T16:01:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55850K .......... .......... .......... .......... .......... 33% 45.7M 4s 2024-04-24T16:01:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55900K .......... .......... .......... .......... .......... 33% 24.9M 4s 2024-04-24T16:01:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 55950K .......... .......... .......... .......... .......... 33% 31.1M 4s 2024-04-24T16:01:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56000K .......... .......... .......... .......... .......... 33% 26.6M 4s 2024-04-24T16:01:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56050K .......... .......... .......... .......... .......... 33% 28.9M 4s 2024-04-24T16:01:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56100K .......... .......... .......... .......... .......... 33% 37.4M 4s 2024-04-24T16:01:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56150K .......... .......... .......... .......... .......... 33% 21.6M 4s 2024-04-24T16:01:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56200K .......... .......... .......... .......... .......... 33% 25.9M 4s 2024-04-24T16:01:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56250K .......... .......... .......... .......... .......... 33% 22.0M 4s 2024-04-24T16:01:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56300K .......... .......... .......... .......... .......... 33% 49.3M 4s 2024-04-24T16:01:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56350K .......... .......... .......... .......... .......... 33% 23.3M 4s 2024-04-24T16:01:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56400K .......... .......... .......... .......... .......... 33% 53.2M 4s 2024-04-24T16:01:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56450K .......... .......... .......... .......... .......... 33% 32.7M 4s 2024-04-24T16:01:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56500K .......... .......... .......... .......... .......... 33% 23.5M 4s 2024-04-24T16:01:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56550K .......... .......... .......... .......... .......... 33% 29.2M 4s 2024-04-24T16:01:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56600K .......... .......... .......... .......... .......... 33% 17.3M 4s 2024-04-24T16:01:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56650K .......... .......... .......... .......... .......... 33% 33.8M 4s 2024-04-24T16:01:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56700K .......... .......... .......... .......... .......... 33% 55.2M 4s 2024-04-24T16:01:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56750K .......... .......... .......... .......... .......... 33% 30.7M 4s 2024-04-24T16:01:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56800K .......... .......... .......... .......... .......... 33% 21.5M 4s 2024-04-24T16:01:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56850K .......... .......... .......... .......... .......... 33% 17.8M 4s 2024-04-24T16:01:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56900K .......... .......... .......... .......... .......... 33% 29.3M 4s 2024-04-24T16:01:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 56950K .......... .......... .......... .......... .......... 33% 12.3M 4s 2024-04-24T16:01:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57000K .......... .......... .......... .......... .......... 33% 36.6M 4s 2024-04-24T16:01:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57050K .......... .......... .......... .......... .......... 33% 28.7M 4s 2024-04-24T16:01:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57100K .......... .......... .......... .......... .......... 33% 57.5M 4s 2024-04-24T16:01:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57150K .......... .......... .......... .......... .......... 33% 19.1M 4s 2024-04-24T16:01:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57200K .......... .......... .......... .......... .......... 33% 11.6M 4s 2024-04-24T16:01:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57250K .......... .......... .......... .......... .......... 33% 24.1M 4s 2024-04-24T16:01:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57300K .......... .......... .......... .......... .......... 34% 14.1M 4s 2024-04-24T16:01:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57350K .......... .......... .......... .......... .......... 34% 11.6M 4s 2024-04-24T16:01:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57400K .......... .......... .......... .......... .......... 34% 23.3M 4s 2024-04-24T16:01:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57450K .......... .......... .......... .......... .......... 34% 30.6M 4s 2024-04-24T16:01:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57500K .......... .......... .......... .......... .......... 34% 44.0M 4s 2024-04-24T16:01:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57550K .......... .......... .......... .......... .......... 34% 12.3M 4s 2024-04-24T16:01:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57600K .......... .......... .......... .......... .......... 34% 10.5M 4s 2024-04-24T16:01:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57650K .......... .......... .......... .......... .......... 34% 43.2M 4s 2024-04-24T16:01:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57700K .......... .......... .......... .......... .......... 34% 12.7M 4s 2024-04-24T16:01:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57750K .......... .......... .......... .......... .......... 34% 21.9M 4s 2024-04-24T16:01:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57800K .......... .......... .......... .......... .......... 34% 21.9M 4s 2024-04-24T16:01:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57850K .......... .......... .......... .......... .......... 34% 110M 4s 2024-04-24T16:01:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57900K .......... .......... .......... .......... .......... 34% 83.2M 4s 2024-04-24T16:01:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 57950K .......... .......... .......... .......... .......... 34% 68.4M 4s 2024-04-24T16:01:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58000K .......... .......... .......... .......... .......... 34% 17.1M 4s 2024-04-24T16:01:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58050K .......... .......... .......... .......... .......... 34% 43.8M 4s 2024-04-24T16:01:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58100K .......... .......... .......... .......... .......... 34% 36.7M 4s 2024-04-24T16:01:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58150K .......... .......... .......... .......... .......... 34% 69.4M 4s 2024-04-24T16:01:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58200K .......... .......... .......... .......... .......... 34% 110M 4s 2024-04-24T16:01:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58250K .......... .......... .......... .......... .......... 34% 92.5M 4s 2024-04-24T16:01:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58300K .......... .......... .......... .......... .......... 34% 110M 4s 2024-04-24T16:01:14.144 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58350K .......... .......... .......... .......... .......... 34% 79.5M 4s 2024-04-24T16:01:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58400K .......... .......... .......... .......... .......... 34% 18.3M 4s 2024-04-24T16:01:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58450K .......... .......... .......... .......... .......... 34% 42.2M 4s 2024-04-24T16:01:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58500K .......... .......... .......... .......... .......... 34% 111M 4s 2024-04-24T16:01:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58550K .......... .......... .......... .......... .......... 34% 94.0M 4s 2024-04-24T16:01:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58600K .......... .......... .......... .......... .......... 34% 75.3M 4s 2024-04-24T16:01:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58650K .......... .......... .......... .......... .......... 34% 22.6M 4s 2024-04-24T16:01:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58700K .......... .......... .......... .......... .......... 34% 45.3M 4s 2024-04-24T16:01:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58750K .......... .......... .......... .......... .......... 34% 26.9M 4s 2024-04-24T16:01:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58800K .......... .......... .......... .......... .......... 34% 30.3M 4s 2024-04-24T16:01:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58850K .......... .......... .......... .......... .......... 34% 49.1M 4s 2024-04-24T16:01:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58900K .......... .......... .......... .......... .......... 34% 52.3M 4s 2024-04-24T16:01:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 58950K .......... .......... .......... .......... .......... 35% 60.7M 4s 2024-04-24T16:01:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59000K .......... .......... .......... .......... .......... 35% 10.6M 4s 2024-04-24T16:01:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59050K .......... .......... .......... .......... .......... 35% 250M 4s 2024-04-24T16:01:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59100K .......... .......... .......... .......... .......... 35% 165M 4s 2024-04-24T16:01:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59150K .......... .......... .......... .......... .......... 35% 78.4M 4s 2024-04-24T16:01:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59200K .......... .......... .......... .......... .......... 35% 63.4M 4s 2024-04-24T16:01:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59250K .......... .......... .......... .......... .......... 35% 45.4M 4s 2024-04-24T16:01:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59300K .......... .......... .......... .......... .......... 35% 48.1M 4s 2024-04-24T16:01:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59350K .......... .......... .......... .......... .......... 35% 28.5M 4s 2024-04-24T16:01:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59400K .......... .......... .......... .......... .......... 35% 58.9M 4s 2024-04-24T16:01:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59450K .......... .......... .......... .......... .......... 35% 45.1M 4s 2024-04-24T16:01:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59500K .......... .......... .......... .......... .......... 35% 37.3M 4s 2024-04-24T16:01:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59550K .......... .......... .......... .......... .......... 35% 11.7M 4s 2024-04-24T16:01:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59600K .......... .......... .......... .......... .......... 35% 45.1M 4s 2024-04-24T16:01:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59650K .......... .......... .......... .......... .......... 35% 29.4M 4s 2024-04-24T16:01:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59700K .......... .......... .......... .......... .......... 35% 49.1M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59750K .......... .......... .......... .......... .......... 35% 40.6M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59800K .......... .......... .......... .......... .......... 35% 38.8M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59850K .......... .......... .......... .......... .......... 35% 20.7M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59900K .......... .......... .......... .......... .......... 35% 42.6M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 59950K .......... .......... .......... .......... .......... 35% 25.5M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60000K .......... .......... .......... .......... .......... 35% 23.4M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60050K .......... .......... .......... .......... .......... 35% 28.0M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60100K .......... .......... .......... .......... .......... 35% 41.6M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60150K .......... .......... .......... .......... .......... 35% 19.5M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60200K .......... .......... .......... .......... .......... 35% 42.5M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60250K .......... .......... .......... .......... .......... 35% 30.3M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60300K .......... .......... .......... .......... .......... 35% 43.8M 4s 2024-04-24T16:01:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60350K .......... .......... .......... .......... .......... 35% 23.3M 4s 2024-04-24T16:01:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60400K .......... .......... .......... .......... .......... 35% 24.6M 4s 2024-04-24T16:01:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60450K .......... .......... .......... .......... .......... 35% 41.1M 4s 2024-04-24T16:01:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60500K .......... .......... .......... .......... .......... 35% 23.6M 4s 2024-04-24T16:01:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60550K .......... .......... .......... .......... .......... 35% 31.8M 4s 2024-04-24T16:01:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60600K .......... .......... .......... .......... .......... 35% 26.2M 4s 2024-04-24T16:01:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60650K .......... .......... .......... .......... .......... 36% 44.8M 4s 2024-04-24T16:01:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60700K .......... .......... .......... .......... .......... 36% 22.2M 4s 2024-04-24T16:01:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60750K .......... .......... .......... .......... .......... 36% 25.5M 4s 2024-04-24T16:01:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60800K .......... .......... .......... .......... .......... 36% 25.2M 4s 2024-04-24T16:01:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60850K .......... .......... .......... .......... .......... 36% 34.0M 4s 2024-04-24T16:01:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60900K .......... .......... .......... .......... .......... 36% 26.1M 4s 2024-04-24T16:01:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 60950K .......... .......... .......... .......... .......... 36% 43.5M 4s 2024-04-24T16:01:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61000K .......... .......... .......... .......... .......... 36% 22.6M 4s 2024-04-24T16:01:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61050K .......... .......... .......... .......... .......... 36% 45.1M 4s 2024-04-24T16:01:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61100K .......... .......... .......... .......... .......... 36% 21.7M 4s 2024-04-24T16:01:14.227 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61150K .......... .......... .......... .......... .......... 36% 24.9M 4s 2024-04-24T16:01:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61200K .......... .......... .......... .......... .......... 36% 35.3M 4s 2024-04-24T16:01:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61250K .......... .......... .......... .......... .......... 36% 23.3M 4s 2024-04-24T16:01:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61300K .......... .......... .......... .......... .......... 36% 49.6M 4s 2024-04-24T16:01:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61350K .......... .......... .......... .......... .......... 36% 22.7M 4s 2024-04-24T16:01:14.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61400K .......... .......... .......... .......... .......... 36% 22.2M 4s 2024-04-24T16:01:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61450K .......... .......... .......... .......... .......... 36% 45.9M 4s 2024-04-24T16:01:14.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61500K .......... .......... .......... .......... .......... 36% 47.6M 4s 2024-04-24T16:01:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61550K .......... .......... .......... .......... .......... 36% 17.2M 4s 2024-04-24T16:01:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61600K .......... .......... .......... .......... .......... 36% 34.5M 4s 2024-04-24T16:01:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61650K .......... .......... .......... .......... .......... 36% 27.1M 4s 2024-04-24T16:01:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61700K .......... .......... .......... .......... .......... 36% 50.5M 4s 2024-04-24T16:01:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61750K .......... .......... .......... .......... .......... 36% 19.9M 4s 2024-04-24T16:01:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61800K .......... .......... .......... .......... .......... 36% 40.9M 4s 2024-04-24T16:01:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61850K .......... .......... .......... .......... .......... 36% 28.1M 4s 2024-04-24T16:01:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61900K .......... .......... .......... .......... .......... 36% 22.2M 4s 2024-04-24T16:01:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 61950K .......... .......... .......... .......... .......... 36% 23.3M 4s 2024-04-24T16:01:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62000K .......... .......... .......... .......... .......... 36% 76.4M 4s 2024-04-24T16:01:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62050K .......... .......... .......... .......... .......... 36% 28.5M 4s 2024-04-24T16:01:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62100K .......... .......... .......... .......... .......... 36% 21.6M 4s 2024-04-24T16:01:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62150K .......... .......... .......... .......... .......... 36% 29.5M 4s 2024-04-24T16:01:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62200K .......... .......... .......... .......... .......... 36% 49.9M 4s 2024-04-24T16:01:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62250K .......... .......... .......... .......... .......... 36% 21.8M 4s 2024-04-24T16:01:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62300K .......... .......... .......... .......... .......... 36% 35.0M 4s 2024-04-24T16:01:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62350K .......... .......... .......... .......... .......... 37% 20.2M 4s 2024-04-24T16:01:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62400K .......... .......... .......... .......... .......... 37% 45.4M 4s 2024-04-24T16:01:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62450K .......... .......... .......... .......... .......... 37% 26.2M 4s 2024-04-24T16:01:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62500K .......... .......... .......... .......... .......... 37% 40.1M 4s 2024-04-24T16:01:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62550K .......... .......... .......... .......... .......... 37% 26.0M 4s 2024-04-24T16:01:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62600K .......... .......... .......... .......... .......... 37% 41.5M 4s 2024-04-24T16:01:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62650K .......... .......... .......... .......... .......... 37% 20.2M 4s 2024-04-24T16:01:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62700K .......... .......... .......... .......... .......... 37% 70.3M 4s 2024-04-24T16:01:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62750K .......... .......... .......... .......... .......... 37% 21.4M 4s 2024-04-24T16:01:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62800K .......... .......... .......... .......... .......... 37% 24.1M 4s 2024-04-24T16:01:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62850K .......... .......... .......... .......... .......... 37% 33.4M 4s 2024-04-24T16:01:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62900K .......... .......... .......... .......... .......... 37% 34.5M 4s 2024-04-24T16:01:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 62950K .......... .......... .......... .......... .......... 37% 26.4M 4s 2024-04-24T16:01:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63000K .......... .......... .......... .......... .......... 37% 31.2M 4s 2024-04-24T16:01:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63050K .......... .......... .......... .......... .......... 37% 22.5M 4s 2024-04-24T16:01:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63100K .......... .......... .......... .......... .......... 37% 35.7M 4s 2024-04-24T16:01:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63150K .......... .......... .......... .......... .......... 37% 24.9M 4s 2024-04-24T16:01:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63200K .......... .......... .......... .......... .......... 37% 24.9M 4s 2024-04-24T16:01:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63250K .......... .......... .......... .......... .......... 37% 36.9M 4s 2024-04-24T16:01:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63300K .......... .......... .......... .......... .......... 37% 29.8M 4s 2024-04-24T16:01:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63350K .......... .......... .......... .......... .......... 37% 37.2M 4s 2024-04-24T16:01:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63400K .......... .......... .......... .......... .......... 37% 22.3M 4s 2024-04-24T16:01:14.304 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63450K .......... .......... .......... .......... .......... 37% 45.2M 4s 2024-04-24T16:01:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63500K .......... .......... .......... .......... .......... 37% 23.8M 4s 2024-04-24T16:01:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63550K .......... .......... .......... .......... .......... 37% 20.7M 4s 2024-04-24T16:01:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63600K .......... .......... .......... .......... .......... 37% 15.8M 4s 2024-04-24T16:01:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63650K .......... .......... .......... .......... .......... 37% 56.8M 4s 2024-04-24T16:01:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63700K .......... .......... .......... .......... .......... 37% 30.2M 4s 2024-04-24T16:01:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63750K .......... .......... .......... .......... .......... 37% 46.3M 4s 2024-04-24T16:01:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63800K .......... .......... .......... .......... .......... 37% 32.3M 4s 2024-04-24T16:01:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63850K .......... .......... .......... .......... .......... 37% 53.9M 4s 2024-04-24T16:01:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63900K .......... .......... .......... .......... .......... 37% 23.5M 4s 2024-04-24T16:01:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 63950K .......... .......... .......... .......... .......... 37% 22.4M 4s 2024-04-24T16:01:14.323 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64000K .......... .......... .......... .......... .......... 37% 43.8M 4s 2024-04-24T16:01:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64050K .......... .......... .......... .......... .......... 38% 22.8M 4s 2024-04-24T16:01:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64100K .......... .......... .......... .......... .......... 38% 43.0M 4s 2024-04-24T16:01:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64150K .......... .......... .......... .......... .......... 38% 25.6M 4s 2024-04-24T16:01:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64200K .......... .......... .......... .......... .......... 38% 19.4M 4s 2024-04-24T16:01:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64250K .......... .......... .......... .......... .......... 38% 105M 4s 2024-04-24T16:01:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64300K .......... .......... .......... .......... .......... 38% 21.2M 4s 2024-04-24T16:01:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64350K .......... .......... .......... .......... .......... 38% 22.5M 4s 2024-04-24T16:01:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64400K .......... .......... .......... .......... .......... 38% 41.8M 4s 2024-04-24T16:01:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64450K .......... .......... .......... .......... .......... 38% 25.7M 4s 2024-04-24T16:01:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64500K .......... .......... .......... .......... .......... 38% 42.4M 4s 2024-04-24T16:01:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64550K .......... .......... .......... .......... .......... 38% 25.8M 4s 2024-04-24T16:01:14.343 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64600K .......... .......... .......... .......... .......... 38% 35.4M 4s 2024-04-24T16:01:14.346 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64650K .......... .......... .......... .......... .......... 38% 19.7M 4s 2024-04-24T16:01:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64700K .......... .......... .......... .......... .......... 38% 32.6M 4s 2024-04-24T16:01:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64750K .......... .......... .......... .......... .......... 38% 23.3M 4s 2024-04-24T16:01:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64800K .......... .......... .......... .......... .......... 38% 42.7M 4s 2024-04-24T16:01:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64850K .......... .......... .......... .......... .......... 38% 25.2M 4s 2024-04-24T16:01:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64900K .......... .......... .......... .......... .......... 38% 36.4M 4s 2024-04-24T16:01:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 64950K .......... .......... .......... .......... .......... 38% 24.5M 4s 2024-04-24T16:01:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65000K .......... .......... .......... .......... .......... 38% 48.5M 4s 2024-04-24T16:01:14.358 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65050K .......... .......... .......... .......... .......... 38% 23.1M 4s 2024-04-24T16:01:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65100K .......... .......... .......... .......... .......... 38% 39.6M 4s 2024-04-24T16:01:14.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65150K .......... .......... .......... .......... .......... 38% 21.6M 4s 2024-04-24T16:01:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65200K .......... .......... .......... .......... .......... 38% 25.9M 4s 2024-04-24T16:01:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65250K .......... .......... .......... .......... .......... 38% 25.0M 4s 2024-04-24T16:01:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65300K .......... .......... .......... .......... .......... 38% 60.5M 4s 2024-04-24T16:01:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65350K .......... .......... .......... .......... .......... 38% 29.8M 4s 2024-04-24T16:01:14.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65400K .......... .......... .......... .......... .......... 38% 26.5M 4s 2024-04-24T16:01:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65450K .......... .......... .......... .......... .......... 38% 20.8M 4s 2024-04-24T16:01:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65500K .......... .......... .......... .......... .......... 38% 47.6M 4s 2024-04-24T16:01:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65550K .......... .......... .......... .......... .......... 38% 25.6M 4s 2024-04-24T16:01:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65600K .......... .......... .......... .......... .......... 38% 35.4M 4s 2024-04-24T16:01:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65650K .......... .......... .......... .......... .......... 38% 25.7M 4s 2024-04-24T16:01:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65700K .......... .......... .......... .......... .......... 39% 27.3M 4s 2024-04-24T16:01:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65750K .......... .......... .......... .......... .......... 39% 45.8M 4s 2024-04-24T16:01:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65800K .......... .......... .......... .......... .......... 39% 20.8M 4s 2024-04-24T16:01:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65850K .......... .......... .......... .......... .......... 39% 28.4M 4s 2024-04-24T16:01:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65900K .......... .......... .......... .......... .......... 39% 54.6M 4s 2024-04-24T16:01:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 65950K .......... .......... .......... .......... .......... 39% 18.2M 4s 2024-04-24T16:01:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66000K .......... .......... .......... .......... .......... 39% 27.3M 4s 2024-04-24T16:01:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66050K .......... .......... .......... .......... .......... 39% 35.1M 4s 2024-04-24T16:01:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66100K .......... .......... .......... .......... .......... 39% 18.6M 4s 2024-04-24T16:01:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66150K .......... .......... .......... .......... .......... 39% 42.0M 4s 2024-04-24T16:01:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66200K .......... .......... .......... .......... .......... 39% 34.4M 4s 2024-04-24T16:01:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66250K .......... .......... .......... .......... .......... 39% 40.1M 4s 2024-04-24T16:01:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66300K .......... .......... .......... .......... .......... 39% 34.0M 4s 2024-04-24T16:01:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66350K .......... .......... .......... .......... .......... 39% 21.3M 4s 2024-04-24T16:01:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66400K .......... .......... .......... .......... .......... 39% 16.8M 4s 2024-04-24T16:01:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66450K .......... .......... .......... .......... .......... 39% 51.7M 4s 2024-04-24T16:01:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66500K .......... .......... .......... .......... .......... 39% 27.6M 4s 2024-04-24T16:01:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66550K .......... .......... .......... .......... .......... 39% 42.8M 4s 2024-04-24T16:01:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66600K .......... .......... .......... .......... .......... 39% 34.6M 4s 2024-04-24T16:01:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66650K .......... .......... .......... .......... .......... 39% 31.6M 4s 2024-04-24T16:01:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66700K .......... .......... .......... .......... .......... 39% 30.1M 4s 2024-04-24T16:01:14.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66750K .......... .......... .......... .......... .......... 39% 17.1M 4s 2024-04-24T16:01:14.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66800K .......... .......... .......... .......... .......... 39% 78.3M 4s 2024-04-24T16:01:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66850K .......... .......... .......... .......... .......... 39% 25.8M 4s 2024-04-24T16:01:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66900K .......... .......... .......... .......... .......... 39% 26.1M 4s 2024-04-24T16:01:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 66950K .......... .......... .......... .......... .......... 39% 30.2M 4s 2024-04-24T16:01:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67000K .......... .......... .......... .......... .......... 39% 25.6M 4s 2024-04-24T16:01:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67050K .......... .......... .......... .......... .......... 39% 22.3M 4s 2024-04-24T16:01:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67100K .......... .......... .......... .......... .......... 39% 48.7M 4s 2024-04-24T16:01:14.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67150K .......... .......... .......... .......... .......... 39% 26.3M 4s 2024-04-24T16:01:14.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67200K .......... .......... .......... .......... .......... 39% 31.7M 4s 2024-04-24T16:01:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67250K .......... .......... .......... .......... .......... 39% 23.1M 4s 2024-04-24T16:01:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67300K .......... .......... .......... .......... .......... 39% 70.2M 4s 2024-04-24T16:01:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67350K .......... .......... .......... .......... .......... 39% 18.1M 4s 2024-04-24T16:01:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67400K .......... .......... .......... .......... .......... 40% 45.8M 4s 2024-04-24T16:01:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67450K .......... .......... .......... .......... .......... 40% 27.2M 4s 2024-04-24T16:01:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67500K .......... .......... .......... .......... .......... 40% 29.8M 4s 2024-04-24T16:01:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67550K .......... .......... .......... .......... .......... 40% 25.8M 4s 2024-04-24T16:01:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67600K .......... .......... .......... .......... .......... 40% 33.8M 4s 2024-04-24T16:01:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67650K .......... .......... .......... .......... .......... 40% 18.0M 4s 2024-04-24T16:01:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67700K .......... .......... .......... .......... .......... 40% 30.5M 4s 2024-04-24T16:01:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67750K .......... .......... .......... .......... .......... 40% 48.0M 4s 2024-04-24T16:01:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67800K .......... .......... .......... .......... .......... 40% 44.2M 4s 2024-04-24T16:01:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67850K .......... .......... .......... .......... .......... 40% 25.7M 4s 2024-04-24T16:01:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67900K .......... .......... .......... .......... .......... 40% 34.9M 4s 2024-04-24T16:01:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 67950K .......... .......... .......... .......... .......... 40% 24.0M 4s 2024-04-24T16:01:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68000K .......... .......... .......... .......... .......... 40% 17.9M 4s 2024-04-24T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68050K .......... .......... .......... .......... .......... 40% 27.7M 4s 2024-04-24T16:01:14.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68100K .......... .......... .......... .......... .......... 40% 64.8M 4s 2024-04-24T16:01:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68150K .......... .......... .......... .......... .......... 40% 28.4M 4s 2024-04-24T16:01:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68200K .......... .......... .......... .......... .......... 40% 38.0M 4s 2024-04-24T16:01:14.466 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68250K .......... .......... .......... .......... .......... 40% 32.2M 4s 2024-04-24T16:01:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68300K .......... .......... .......... .......... .......... 40% 34.2M 4s 2024-04-24T16:01:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68350K .......... .......... .......... .......... .......... 40% 15.6M 4s 2024-04-24T16:01:14.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68400K .......... .......... .......... .......... .......... 40% 28.2M 4s 2024-04-24T16:01:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68450K .......... .......... .......... .......... .......... 40% 58.1M 4s 2024-04-24T16:01:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68500K .......... .......... .......... .......... .......... 40% 25.9M 4s 2024-04-24T16:01:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68550K .......... .......... .......... .......... .......... 40% 47.6M 4s 2024-04-24T16:01:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68600K .......... .......... .......... .......... .......... 40% 24.9M 4s 2024-04-24T16:01:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68650K .......... .......... .......... .......... .......... 40% 18.5M 4s 2024-04-24T16:01:14.483 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68700K .......... .......... .......... .......... .......... 40% 43.1M 4s 2024-04-24T16:01:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68750K .......... .......... .......... .......... .......... 40% 25.8M 4s 2024-04-24T16:01:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68800K .......... .......... .......... .......... .......... 40% 49.0M 4s 2024-04-24T16:01:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68850K .......... .......... .......... .......... .......... 40% 28.3M 4s 2024-04-24T16:01:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68900K .......... .......... .......... .......... .......... 40% 29.6M 4s 2024-04-24T16:01:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 68950K .......... .......... .......... .......... .......... 40% 19.4M 4s 2024-04-24T16:01:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69000K .......... .......... .......... .......... .......... 40% 29.4M 4s 2024-04-24T16:01:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69050K .......... .......... .......... .......... .......... 40% 51.7M 4s 2024-04-24T16:01:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69100K .......... .......... .......... .......... .......... 41% 30.6M 4s 2024-04-24T16:01:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69150K .......... .......... .......... .......... .......... 41% 20.4M 4s 2024-04-24T16:01:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69200K .......... .......... .......... .......... .......... 41% 29.1M 4s 2024-04-24T16:01:14.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69250K .......... .......... .......... .......... .......... 41% 25.9M 4s 2024-04-24T16:01:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69300K .......... .......... .......... .......... .......... 41% 41.7M 4s 2024-04-24T16:01:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69350K .......... .......... .......... .......... .......... 41% 25.0M 4s 2024-04-24T16:01:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69400K .......... .......... .......... .......... .......... 41% 26.0M 4s 2024-04-24T16:01:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69450K .......... .......... .......... .......... .......... 41% 42.5M 4s 2024-04-24T16:01:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69500K .......... .......... .......... .......... .......... 41% 21.8M 4s 2024-04-24T16:01:14.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69550K .......... .......... .......... .......... .......... 41% 19.0M 4s 2024-04-24T16:01:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69600K .......... .......... .......... .......... .......... 41% 46.0M 4s 2024-04-24T16:01:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69650K .......... .......... .......... .......... .......... 41% 40.6M 4s 2024-04-24T16:01:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69700K .......... .......... .......... .......... .......... 41% 22.7M 4s 2024-04-24T16:01:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69750K .......... .......... .......... .......... .......... 41% 31.8M 4s 2024-04-24T16:01:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69800K .......... .......... .......... .......... .......... 41% 34.2M 4s 2024-04-24T16:01:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69850K .......... .......... .......... .......... .......... 41% 33.5M 4s 2024-04-24T16:01:14.523 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69900K .......... .......... .......... .......... .......... 41% 21.5M 4s 2024-04-24T16:01:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 69950K .......... .......... .......... .......... .......... 41% 26.6M 4s 2024-04-24T16:01:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70000K .......... .......... .......... .......... .......... 41% 36.0M 4s 2024-04-24T16:01:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70050K .......... .......... .......... .......... .......... 41% 23.5M 4s 2024-04-24T16:01:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70100K .......... .......... .......... .......... .......... 41% 47.0M 4s 2024-04-24T16:01:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70150K .......... .......... .......... .......... .......... 41% 24.9M 4s 2024-04-24T16:01:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70200K .......... .......... .......... .......... .......... 41% 21.5M 4s 2024-04-24T16:01:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70250K .......... .......... .......... .......... .......... 41% 38.0M 4s 2024-04-24T16:01:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70300K .......... .......... .......... .......... .......... 41% 75.0M 4s 2024-04-24T16:01:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70350K .......... .......... .......... .......... .......... 41% 13.7M 4s 2024-04-24T16:01:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70400K .......... .......... .......... .......... .......... 41% 54.1M 4s 2024-04-24T16:01:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70450K .......... .......... .......... .......... .......... 41% 44.6M 4s 2024-04-24T16:01:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70500K .......... .......... .......... .......... .......... 41% 19.2M 4s 2024-04-24T16:01:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70550K .......... .......... .......... .......... .......... 41% 44.3M 4s 2024-04-24T16:01:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70600K .......... .......... .......... .......... .......... 41% 35.9M 4s 2024-04-24T16:01:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70650K .......... .......... .......... .......... .......... 41% 20.6M 4s 2024-04-24T16:01:14.549 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70700K .......... .......... .......... .......... .......... 41% 37.3M 4s 2024-04-24T16:01:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70750K .......... .......... .......... .......... .......... 42% 28.8M 4s 2024-04-24T16:01:14.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70800K .......... .......... .......... .......... .......... 42% 22.2M 4s 2024-04-24T16:01:14.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70850K .......... .......... .......... .......... .......... 42% 48.0M 4s 2024-04-24T16:01:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70900K .......... .......... .......... .......... .......... 42% 28.1M 4s 2024-04-24T16:01:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 70950K .......... .......... .......... .......... .......... 42% 19.0M 4s 2024-04-24T16:01:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71000K .......... .......... .......... .......... .......... 42% 48.0M 4s 2024-04-24T16:01:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71050K .......... .......... .......... .......... .......... 42% 30.0M 4s 2024-04-24T16:01:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71100K .......... .......... .......... .......... .......... 42% 38.2M 4s 2024-04-24T16:01:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71150K .......... .......... .......... .......... .......... 42% 21.7M 4s 2024-04-24T16:01:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71200K .......... .......... .......... .......... .......... 42% 26.3M 4s 2024-04-24T16:01:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71250K .......... .......... .......... .......... .......... 42% 38.7M 4s 2024-04-24T16:01:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71300K .......... .......... .......... .......... .......... 42% 22.9M 4s 2024-04-24T16:01:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71350K .......... .......... .......... .......... .......... 42% 53.6M 4s 2024-04-24T16:01:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71400K .......... .......... .......... .......... .......... 42% 24.7M 4s 2024-04-24T16:01:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71450K .......... .......... .......... .......... .......... 42% 21.3M 4s 2024-04-24T16:01:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71500K .......... .......... .......... .......... .......... 42% 39.1M 4s 2024-04-24T16:01:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71550K .......... .......... .......... .......... .......... 42% 29.2M 4s 2024-04-24T16:01:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71600K .......... .......... .......... .......... .......... 42% 22.8M 4s 2024-04-24T16:01:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71650K .......... .......... .......... .......... .......... 42% 43.0M 4s 2024-04-24T16:01:14.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71700K .......... .......... .......... .......... .......... 42% 33.2M 4s 2024-04-24T16:01:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71750K .......... .......... .......... .......... .......... 42% 26.5M 4s 2024-04-24T16:01:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71800K .......... .......... .......... .......... .......... 42% 22.0M 4s 2024-04-24T16:01:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71850K .......... .......... .......... .......... .......... 42% 44.6M 4s 2024-04-24T16:01:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71900K .......... .......... .......... .......... .......... 42% 43.2M 4s 2024-04-24T16:01:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 71950K .......... .......... .......... .......... .......... 42% 16.4M 4s 2024-04-24T16:01:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72000K .......... .......... .......... .......... .......... 42% 42.3M 3s 2024-04-24T16:01:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72050K .......... .......... .......... .......... .......... 42% 27.9M 3s 2024-04-24T16:01:14.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72100K .......... .......... .......... .......... .......... 42% 20.0M 3s 2024-04-24T16:01:14.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72150K .......... .......... .......... .......... .......... 42% 38.9M 3s 2024-04-24T16:01:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72200K .......... .......... .......... .......... .......... 42% 37.6M 3s 2024-04-24T16:01:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72250K .......... .......... .......... .......... .......... 42% 25.2M 3s 2024-04-24T16:01:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72300K .......... .......... .......... .......... .......... 42% 27.0M 3s 2024-04-24T16:01:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72350K .......... .......... .......... .......... .......... 42% 25.9M 3s 2024-04-24T16:01:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72400K .......... .......... .......... .......... .......... 42% 45.4M 3s 2024-04-24T16:01:14.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72450K .......... .......... .......... .......... .......... 43% 24.6M 3s 2024-04-24T16:01:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72500K .......... .......... .......... .......... .......... 43% 34.8M 3s 2024-04-24T16:01:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72550K .......... .......... .......... .......... .......... 43% 24.2M 3s 2024-04-24T16:01:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72600K .......... .......... .......... .......... .......... 43% 25.9M 3s 2024-04-24T16:01:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72650K .......... .......... .......... .......... .......... 43% 33.1M 3s 2024-04-24T16:01:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72700K .......... .......... .......... .......... .......... 43% 33.1M 3s 2024-04-24T16:01:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72750K .......... .......... .......... .......... .......... 43% 22.7M 3s 2024-04-24T16:01:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72800K .......... .......... .......... .......... .......... 43% 36.0M 3s 2024-04-24T16:01:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72850K .......... .......... .......... .......... .......... 43% 18.9M 3s 2024-04-24T16:01:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72900K .......... .......... .......... .......... .......... 43% 46.2M 3s 2024-04-24T16:01:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 72950K .......... .......... .......... .......... .......... 43% 41.9M 3s 2024-04-24T16:01:14.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73000K .......... .......... .......... .......... .......... 43% 26.7M 3s 2024-04-24T16:01:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73050K .......... .......... .......... .......... .......... 43% 27.9M 3s 2024-04-24T16:01:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73100K .......... .......... .......... .......... .......... 43% 46.9M 3s 2024-04-24T16:01:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73150K .......... .......... .......... .......... .......... 43% 15.4M 3s 2024-04-24T16:01:14.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73200K .......... .......... .......... .......... .......... 43% 34.9M 3s 2024-04-24T16:01:14.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73250K .......... .......... .......... .......... .......... 43% 29.1M 3s 2024-04-24T16:01:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73300K .......... .......... .......... .......... .......... 43% 38.2M 3s 2024-04-24T16:01:14.640 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73350K .......... .......... .......... .......... .......... 43% 26.3M 3s 2024-04-24T16:01:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73400K .......... .......... .......... .......... .......... 43% 47.8M 3s 2024-04-24T16:01:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73450K .......... .......... .......... .......... .......... 43% 32.9M 3s 2024-04-24T16:01:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73500K .......... .......... .......... .......... .......... 43% 30.9M 3s 2024-04-24T16:01:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73550K .......... .......... .......... .......... .......... 43% 20.1M 3s 2024-04-24T16:01:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73600K .......... .......... .......... .......... .......... 43% 32.1M 3s 2024-04-24T16:01:14.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73650K .......... .......... .......... .......... .......... 43% 33.6M 3s 2024-04-24T16:01:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73700K .......... .......... .......... .......... .......... 43% 23.6M 3s 2024-04-24T16:01:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73750K .......... .......... .......... .......... .......... 43% 32.6M 3s 2024-04-24T16:01:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73800K .......... .......... .......... .......... .......... 43% 23.8M 3s 2024-04-24T16:01:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73850K .......... .......... .......... .......... .......... 43% 30.5M 3s 2024-04-24T16:01:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73900K .......... .......... .......... .......... .......... 43% 30.5M 3s 2024-04-24T16:01:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 73950K .......... .......... .......... .......... .......... 43% 36.2M 3s 2024-04-24T16:01:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74000K .......... .......... .......... .......... .......... 43% 20.6M 3s 2024-04-24T16:01:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74050K .......... .......... .......... .......... .......... 43% 29.5M 3s 2024-04-24T16:01:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74100K .......... .......... .......... .......... .......... 43% 61.4M 3s 2024-04-24T16:01:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74150K .......... .......... .......... .......... .......... 44% 27.8M 3s 2024-04-24T16:01:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74200K .......... .......... .......... .......... .......... 44% 25.3M 3s 2024-04-24T16:01:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74250K .......... .......... .......... .......... .......... 44% 28.1M 3s 2024-04-24T16:01:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74300K .......... .......... .......... .......... .......... 44% 28.4M 3s 2024-04-24T16:01:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74350K .......... .......... .......... .......... .......... 44% 23.2M 3s 2024-04-24T16:01:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74400K .......... .......... .......... .......... .......... 44% 25.2M 3s 2024-04-24T16:01:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74450K .......... .......... .......... .......... .......... 44% 51.8M 3s 2024-04-24T16:01:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74500K .......... .......... .......... .......... .......... 44% 24.9M 3s 2024-04-24T16:01:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74550K .......... .......... .......... .......... .......... 44% 32.4M 3s 2024-04-24T16:01:14.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74600K .......... .......... .......... .......... .......... 44% 25.1M 3s 2024-04-24T16:01:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74650K .......... .......... .......... .......... .......... 44% 35.5M 3s 2024-04-24T16:01:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74700K .......... .......... .......... .......... .......... 44% 25.8M 3s 2024-04-24T16:01:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74750K .......... .......... .......... .......... .......... 44% 25.4M 3s 2024-04-24T16:01:14.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74800K .......... .......... .......... .......... .......... 44% 40.1M 3s 2024-04-24T16:01:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74850K .......... .......... .......... .......... .......... 44% 22.8M 3s 2024-04-24T16:01:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74900K .......... .......... .......... .......... .......... 44% 27.0M 3s 2024-04-24T16:01:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 74950K .......... .......... .......... .......... .......... 44% 32.4M 3s 2024-04-24T16:01:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75000K .......... .......... .......... .......... .......... 44% 26.4M 3s 2024-04-24T16:01:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75050K .......... .......... .......... .......... .......... 44% 36.6M 3s 2024-04-24T16:01:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75100K .......... .......... .......... .......... .......... 44% 25.2M 3s 2024-04-24T16:01:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75150K .......... .......... .......... .......... .......... 44% 23.2M 3s 2024-04-24T16:01:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75200K .......... .......... .......... .......... .......... 44% 59.8M 3s 2024-04-24T16:01:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75250K .......... .......... .......... .......... .......... 44% 23.3M 3s 2024-04-24T16:01:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75300K .......... .......... .......... .......... .......... 44% 35.9M 3s 2024-04-24T16:01:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75350K .......... .......... .......... .......... .......... 44% 23.5M 3s 2024-04-24T16:01:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75400K .......... .......... .......... .......... .......... 44% 38.6M 3s 2024-04-24T16:01:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75450K .......... .......... .......... .......... .......... 44% 27.9M 3s 2024-04-24T16:01:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75500K .......... .......... .......... .......... .......... 44% 33.3M 3s 2024-04-24T16:01:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75550K .......... .......... .......... .......... .......... 44% 19.1M 3s 2024-04-24T16:01:14.715 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75600K .......... .......... .......... .......... .......... 44% 37.5M 3s 2024-04-24T16:01:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75650K .......... .......... .......... .......... .......... 44% 27.8M 3s 2024-04-24T16:01:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75700K .......... .......... .......... .......... .......... 44% 26.6M 3s 2024-04-24T16:01:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75750K .......... .......... .......... .......... .......... 44% 27.6M 3s 2024-04-24T16:01:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75800K .......... .......... .......... .......... .......... 44% 38.4M 3s 2024-04-24T16:01:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75850K .......... .......... .......... .......... .......... 45% 23.6M 3s 2024-04-24T16:01:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75900K .......... .......... .......... .......... .......... 45% 48.6M 3s 2024-04-24T16:01:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 75950K .......... .......... .......... .......... .......... 45% 22.4M 3s 2024-04-24T16:01:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76000K .......... .......... .......... .......... .......... 45% 18.3M 3s 2024-04-24T16:01:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76050K .......... .......... .......... .......... .......... 45% 43.4M 3s 2024-04-24T16:01:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76100K .......... .......... .......... .......... .......... 45% 46.6M 3s 2024-04-24T16:01:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76150K .......... .......... .......... .......... .......... 45% 26.8M 3s 2024-04-24T16:01:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76200K .......... .......... .......... .......... .......... 45% 28.2M 3s 2024-04-24T16:01:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76250K .......... .......... .......... .......... .......... 45% 24.1M 3s 2024-04-24T16:01:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76300K .......... .......... .......... .......... .......... 45% 49.8M 3s 2024-04-24T16:01:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76350K .......... .......... .......... .......... .......... 45% 20.3M 3s 2024-04-24T16:01:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76400K .......... .......... .......... .......... .......... 45% 30.0M 3s 2024-04-24T16:01:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76450K .......... .......... .......... .......... .......... 45% 42.6M 3s 2024-04-24T16:01:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76500K .......... .......... .......... .......... .......... 45% 20.8M 3s 2024-04-24T16:01:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76550K .......... .......... .......... .......... .......... 45% 37.9M 3s 2024-04-24T16:01:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76600K .......... .......... .......... .......... .......... 45% 23.8M 3s 2024-04-24T16:01:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76650K .......... .......... .......... .......... .......... 45% 38.7M 3s 2024-04-24T16:01:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76700K .......... .......... .......... .......... .......... 45% 29.6M 3s 2024-04-24T16:01:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76750K .......... .......... .......... .......... .......... 45% 24.6M 3s 2024-04-24T16:01:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76800K .......... .......... .......... .......... .......... 45% 21.5M 3s 2024-04-24T16:01:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76850K .......... .......... .......... .......... .......... 45% 60.7M 3s 2024-04-24T16:01:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76900K .......... .......... .......... .......... .......... 45% 20.8M 3s 2024-04-24T16:01:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 76950K .......... .......... .......... .......... .......... 45% 57.1M 3s 2024-04-24T16:01:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77000K .......... .......... .......... .......... .......... 45% 22.6M 3s 2024-04-24T16:01:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77050K .......... .......... .......... .......... .......... 45% 41.3M 3s 2024-04-24T16:01:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77100K .......... .......... .......... .......... .......... 45% 22.1M 3s 2024-04-24T16:01:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77150K .......... .......... .......... .......... .......... 45% 28.7M 3s 2024-04-24T16:01:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77200K .......... .......... .......... .......... .......... 45% 26.6M 3s 2024-04-24T16:01:14.772 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77250K .......... .......... .......... .......... .......... 45% 29.5M 3s 2024-04-24T16:01:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77300K .......... .......... .......... .......... .......... 45% 40.4M 3s 2024-04-24T16:01:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77350K .......... .......... .......... .......... .......... 45% 24.2M 3s 2024-04-24T16:01:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77400K .......... .......... .......... .......... .......... 45% 28.2M 3s 2024-04-24T16:01:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77450K .......... .......... .......... .......... .......... 45% 56.6M 3s 2024-04-24T16:01:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77500K .......... .......... .......... .......... .......... 46% 21.8M 3s 2024-04-24T16:01:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77550K .......... .......... .......... .......... .......... 46% 25.2M 3s 2024-04-24T16:01:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77600K .......... .......... .......... .......... .......... 46% 26.4M 3s 2024-04-24T16:01:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77650K .......... .......... .......... .......... .......... 46% 27.3M 3s 2024-04-24T16:01:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77700K .......... .......... .......... .......... .......... 46% 19.8M 3s 2024-04-24T16:01:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77750K .......... .......... .......... .......... .......... 46% 63.3M 3s 2024-04-24T16:01:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77800K .......... .......... .......... .......... .......... 46% 27.4M 3s 2024-04-24T16:01:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77850K .......... .......... .......... .......... .......... 46% 26.0M 3s 2024-04-24T16:01:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77900K .......... .......... .......... .......... .......... 46% 42.2M 3s 2024-04-24T16:01:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 77950K .......... .......... .......... .......... .......... 46% 17.6M 3s 2024-04-24T16:01:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78000K .......... .......... .......... .......... .......... 46% 61.5M 3s 2024-04-24T16:01:14.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78050K .......... .......... .......... .......... .......... 46% 26.4M 3s 2024-04-24T16:01:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78100K .......... .......... .......... .......... .......... 46% 28.4M 3s 2024-04-24T16:01:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78150K .......... .......... .......... .......... .......... 46% 18.1M 3s 2024-04-24T16:01:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78200K .......... .......... .......... .......... .......... 46% 20.6M 3s 2024-04-24T16:01:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78250K .......... .......... .......... .......... .......... 46% 17.0M 3s 2024-04-24T16:01:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78300K .......... .......... .......... .......... .......... 46% 41.2M 3s 2024-04-24T16:01:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78350K .......... .......... .......... .......... .......... 46% 39.3M 3s 2024-04-24T16:01:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78400K .......... .......... .......... .......... .......... 46% 40.1M 3s 2024-04-24T16:01:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78450K .......... .......... .......... .......... .......... 46% 48.3M 3s 2024-04-24T16:01:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78500K .......... .......... .......... .......... .......... 46% 61.9M 3s 2024-04-24T16:01:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78550K .......... .......... .......... .......... .......... 46% 24.7M 3s 2024-04-24T16:01:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78600K .......... .......... .......... .......... .......... 46% 39.3M 3s 2024-04-24T16:01:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78650K .......... .......... .......... .......... .......... 46% 27.7M 3s 2024-04-24T16:01:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78700K .......... .......... .......... .......... .......... 46% 34.2M 3s 2024-04-24T16:01:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78750K .......... .......... .......... .......... .......... 46% 22.0M 3s 2024-04-24T16:01:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78800K .......... .......... .......... .......... .......... 46% 29.0M 3s 2024-04-24T16:01:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78850K .......... .......... .......... .......... .......... 46% 23.5M 3s 2024-04-24T16:01:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78900K .......... .......... .......... .......... .......... 46% 34.7M 3s 2024-04-24T16:01:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 78950K .......... .......... .......... .......... .......... 46% 26.5M 3s 2024-04-24T16:01:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79000K .......... .......... .......... .......... .......... 46% 39.4M 3s 2024-04-24T16:01:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79050K .......... .......... .......... .......... .......... 46% 24.2M 3s 2024-04-24T16:01:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79100K .......... .......... .......... .......... .......... 46% 52.2M 3s 2024-04-24T16:01:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79150K .......... .......... .......... .......... .......... 46% 20.6M 3s 2024-04-24T16:01:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79200K .......... .......... .......... .......... .......... 47% 27.8M 3s 2024-04-24T16:01:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79250K .......... .......... .......... .......... .......... 47% 38.2M 3s 2024-04-24T16:01:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79300K .......... .......... .......... .......... .......... 47% 24.7M 3s 2024-04-24T16:01:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79350K .......... .......... .......... .......... .......... 47% 37.6M 3s 2024-04-24T16:01:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79400K .......... .......... .......... .......... .......... 47% 23.2M 3s 2024-04-24T16:01:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79450K .......... .......... .......... .......... .......... 47% 23.3M 3s 2024-04-24T16:01:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79500K .......... .......... .......... .......... .......... 47% 58.0M 3s 2024-04-24T16:01:14.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79550K .......... .......... .......... .......... .......... 47% 22.2M 3s 2024-04-24T16:01:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79600K .......... .......... .......... .......... .......... 47% 23.4M 3s 2024-04-24T16:01:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79650K .......... .......... .......... .......... .......... 47% 40.8M 3s 2024-04-24T16:01:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79700K .......... .......... .......... .......... .......... 47% 23.5M 3s 2024-04-24T16:01:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79750K .......... .......... .......... .......... .......... 47% 35.7M 3s 2024-04-24T16:01:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79800K .......... .......... .......... .......... .......... 47% 28.7M 3s 2024-04-24T16:01:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79850K .......... .......... .......... .......... .......... 47% 37.1M 3s 2024-04-24T16:01:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79900K .......... .......... .......... .......... .......... 47% 23.1M 3s 2024-04-24T16:01:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 79950K .......... .......... .......... .......... .......... 47% 26.0M 3s 2024-04-24T16:01:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80000K .......... .......... .......... .......... .......... 47% 24.7M 3s 2024-04-24T16:01:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80050K .......... .......... .......... .......... .......... 47% 39.2M 3s 2024-04-24T16:01:14.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80100K .......... .......... .......... .......... .......... 47% 26.6M 3s 2024-04-24T16:01:14.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80150K .......... .......... .......... .......... .......... 47% 21.9M 3s 2024-04-24T16:01:14.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80200K .......... .......... .......... .......... .......... 47% 41.6M 3s 2024-04-24T16:01:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80250K .......... .......... .......... .......... .......... 47% 33.9M 3s 2024-04-24T16:01:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80300K .......... .......... .......... .......... .......... 47% 27.0M 3s 2024-04-24T16:01:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80350K .......... .......... .......... .......... .......... 47% 31.5M 3s 2024-04-24T16:01:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80400K .......... .......... .......... .......... .......... 47% 26.7M 3s 2024-04-24T16:01:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80450K .......... .......... .......... .......... .......... 47% 30.8M 3s 2024-04-24T16:01:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80500K .......... .......... .......... .......... .......... 47% 37.7M 3s 2024-04-24T16:01:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80550K .......... .......... .......... .......... .......... 47% 27.4M 3s 2024-04-24T16:01:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80600K .......... .......... .......... .......... .......... 47% 37.9M 3s 2024-04-24T16:01:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80650K .......... .......... .......... .......... .......... 47% 22.4M 3s 2024-04-24T16:01:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80700K .......... .......... .......... .......... .......... 47% 30.9M 3s 2024-04-24T16:01:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80750K .......... .......... .......... .......... .......... 47% 21.2M 3s 2024-04-24T16:01:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80800K .......... .......... .......... .......... .......... 47% 34.5M 3s 2024-04-24T16:01:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80850K .......... .......... .......... .......... .......... 47% 29.9M 3s 2024-04-24T16:01:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80900K .......... .......... .......... .......... .......... 48% 36.9M 3s 2024-04-24T16:01:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 80950K .......... .......... .......... .......... .......... 48% 21.6M 3s 2024-04-24T16:01:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81000K .......... .......... .......... .......... .......... 48% 38.4M 3s 2024-04-24T16:01:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81050K .......... .......... .......... .......... .......... 48% 28.3M 3s 2024-04-24T16:01:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81100K .......... .......... .......... .......... .......... 48% 20.5M 3s 2024-04-24T16:01:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81150K .......... .......... .......... .......... .......... 48% 27.6M 3s 2024-04-24T16:01:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81200K .......... .......... .......... .......... .......... 48% 36.0M 3s 2024-04-24T16:01:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81250K .......... .......... .......... .......... .......... 48% 24.5M 3s 2024-04-24T16:01:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81300K .......... .......... .......... .......... .......... 48% 46.7M 3s 2024-04-24T16:01:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81350K .......... .......... .......... .......... .......... 48% 36.5M 3s 2024-04-24T16:01:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81400K .......... .......... .......... .......... .......... 48% 24.8M 3s 2024-04-24T16:01:14.914 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81450K .......... .......... .......... .......... .......... 48% 21.9M 3s 2024-04-24T16:01:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81500K .......... .......... .......... .......... .......... 48% 66.8M 3s 2024-04-24T16:01:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81550K .......... .......... .......... .......... .......... 48% 20.8M 3s 2024-04-24T16:01:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81600K .......... .......... .......... .......... .......... 48% 29.9M 3s 2024-04-24T16:01:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81650K .......... .......... .......... .......... .......... 48% 25.2M 3s 2024-04-24T16:01:14.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81700K .......... .......... .......... .......... .......... 48% 33.3M 3s 2024-04-24T16:01:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81750K .......... .......... .......... .......... .......... 48% 22.2M 3s 2024-04-24T16:01:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81800K .......... .......... .......... .......... .......... 48% 57.2M 3s 2024-04-24T16:01:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81850K .......... .......... .......... .......... .......... 48% 24.4M 3s 2024-04-24T16:01:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81900K .......... .......... .......... .......... .......... 48% 27.0M 3s 2024-04-24T16:01:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 81950K .......... .......... .......... .......... .......... 48% 29.2M 3s 2024-04-24T16:01:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82000K .......... .......... .......... .......... .......... 48% 23.8M 3s 2024-04-24T16:01:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82050K .......... .......... .......... .......... .......... 48% 46.9M 3s 2024-04-24T16:01:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82100K .......... .......... .......... .......... .......... 48% 25.5M 3s 2024-04-24T16:01:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82150K .......... .......... .......... .......... .......... 48% 34.9M 3s 2024-04-24T16:01:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82200K .......... .......... .......... .......... .......... 48% 24.0M 3s 2024-04-24T16:01:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82250K .......... .......... .......... .......... .......... 48% 42.5M 3s 2024-04-24T16:01:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82300K .......... .......... .......... .......... .......... 48% 24.9M 3s 2024-04-24T16:01:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82350K .......... .......... .......... .......... .......... 48% 26.3M 3s 2024-04-24T16:01:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82400K .......... .......... .......... .......... .......... 48% 22.8M 3s 2024-04-24T16:01:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82450K .......... .......... .......... .......... .......... 48% 35.5M 3s 2024-04-24T16:01:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82500K .......... .......... .......... .......... .......... 48% 25.4M 3s 2024-04-24T16:01:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82550K .......... .......... .......... .......... .......... 49% 48.4M 3s 2024-04-24T16:01:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82600K .......... .......... .......... .......... .......... 49% 27.9M 3s 2024-04-24T16:01:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82650K .......... .......... .......... .......... .......... 49% 30.5M 3s 2024-04-24T16:01:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82700K .......... .......... .......... .......... .......... 49% 26.0M 3s 2024-04-24T16:01:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82750K .......... .......... .......... .......... .......... 49% 19.3M 3s 2024-04-24T16:01:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82800K .......... .......... .......... .......... .......... 49% 34.5M 3s 2024-04-24T16:01:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82850K .......... .......... .......... .......... .......... 49% 44.7M 3s 2024-04-24T16:01:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82900K .......... .......... .......... .......... .......... 49% 31.8M 3s 2024-04-24T16:01:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 82950K .......... .......... .......... .......... .......... 49% 29.6M 3s 2024-04-24T16:01:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83000K .......... .......... .......... .......... .......... 49% 20.3M 3s 2024-04-24T16:01:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83050K .......... .......... .......... .......... .......... 49% 53.1M 3s 2024-04-24T16:01:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83100K .......... .......... .......... .......... .......... 49% 27.8M 3s 2024-04-24T16:01:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83150K .......... .......... .......... .......... .......... 49% 21.3M 3s 2024-04-24T16:01:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83200K .......... .......... .......... .......... .......... 49% 25.9M 3s 2024-04-24T16:01:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83250K .......... .......... .......... .......... .......... 49% 45.1M 3s 2024-04-24T16:01:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83300K .......... .......... .......... .......... .......... 49% 36.4M 3s 2024-04-24T16:01:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83350K .......... .......... .......... .......... .......... 49% 22.8M 3s 2024-04-24T16:01:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83400K .......... .......... .......... .......... .......... 49% 27.1M 3s 2024-04-24T16:01:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83450K .......... .......... .......... .......... .......... 49% 32.6M 3s 2024-04-24T16:01:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83500K .......... .......... .......... .......... .......... 49% 26.0M 3s 2024-04-24T16:01:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83550K .......... .......... .......... .......... .......... 49% 22.1M 3s 2024-04-24T16:01:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83600K .......... .......... .......... .......... .......... 49% 58.7M 3s 2024-04-24T16:01:14.988 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83650K .......... .......... .......... .......... .......... 49% 21.2M 3s 2024-04-24T16:01:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83700K .......... .......... .......... .......... .......... 49% 41.6M 3s 2024-04-24T16:01:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83750K .......... .......... .......... .......... .......... 49% 18.7M 3s 2024-04-24T16:01:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83800K .......... .......... .......... .......... .......... 49% 106M 3s 2024-04-24T16:01:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83850K .......... .......... .......... .......... .......... 49% 25.2M 3s 2024-04-24T16:01:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83900K .......... .......... .......... .......... .......... 49% 34.3M 3s 2024-04-24T16:01:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 83950K .......... .......... .......... .......... .......... 49% 16.9M 3s 2024-04-24T16:01:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84000K .......... .......... .......... .......... .......... 49% 39.2M 3s 2024-04-24T16:01:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84050K .......... .......... .......... .......... .......... 49% 23.7M 3s 2024-04-24T16:01:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84100K .......... .......... .......... .......... .......... 49% 43.7M 3s 2024-04-24T16:01:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84150K .......... .......... .......... .......... .......... 49% 24.9M 3s 2024-04-24T16:01:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84200K .......... .......... .......... .......... .......... 49% 36.6M 3s 2024-04-24T16:01:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84250K .......... .......... .......... .......... .......... 50% 19.9M 3s 2024-04-24T16:01:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84300K .......... .......... .......... .......... .......... 50% 56.8M 3s 2024-04-24T16:01:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84350K .......... .......... .......... .......... .......... 50% 25.7M 3s 2024-04-24T16:01:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84400K .......... .......... .......... .......... .......... 50% 30.7M 3s 2024-04-24T16:01:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84450K .......... .......... .......... .......... .......... 50% 30.4M 3s 2024-04-24T16:01:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84500K .......... .......... .......... .......... .......... 50% 27.2M 3s 2024-04-24T16:01:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84550K .......... .......... .......... .......... .......... 50% 38.1M 3s 2024-04-24T16:01:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84600K .......... .......... .......... .......... .......... 50% 19.8M 3s 2024-04-24T16:01:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84650K .......... .......... .......... .......... .......... 50% 33.2M 3s 2024-04-24T16:01:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84700K .......... .......... .......... .......... .......... 50% 55.2M 3s 2024-04-24T16:01:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84750K .......... .......... .......... .......... .......... 50% 17.8M 3s 2024-04-24T16:01:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84800K .......... .......... .......... .......... .......... 50% 31.7M 3s 2024-04-24T16:01:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84850K .......... .......... .......... .......... .......... 50% 41.9M 3s 2024-04-24T16:01:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84900K .......... .......... .......... .......... .......... 50% 25.2M 3s 2024-04-24T16:01:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 84950K .......... .......... .......... .......... .......... 50% 25.0M 3s 2024-04-24T16:01:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85000K .......... .......... .......... .......... .......... 50% 27.9M 3s 2024-04-24T16:01:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85050K .......... .......... .......... .......... .......... 50% 26.9M 3s 2024-04-24T16:01:15.036 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85100K .......... .......... .......... .......... .......... 50% 48.2M 3s 2024-04-24T16:01:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85150K .......... .......... .......... .......... .......... 50% 25.2M 3s 2024-04-24T16:01:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85200K .......... .......... .......... .......... .......... 50% 21.8M 3s 2024-04-24T16:01:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85250K .......... .......... .......... .......... .......... 50% 35.8M 3s 2024-04-24T16:01:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85300K .......... .......... .......... .......... .......... 50% 31.1M 3s 2024-04-24T16:01:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85350K .......... .......... .......... .......... .......... 50% 37.4M 3s 2024-04-24T16:01:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85400K .......... .......... .......... .......... .......... 50% 27.0M 3s 2024-04-24T16:01:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85450K .......... .......... .......... .......... .......... 50% 29.1M 3s 2024-04-24T16:01:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85500K .......... .......... .......... .......... .......... 50% 23.4M 3s 2024-04-24T16:01:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85550K .......... .......... .......... .......... .......... 50% 29.1M 3s 2024-04-24T16:01:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85600K .......... .......... .......... .......... .......... 50% 24.9M 3s 2024-04-24T16:01:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85650K .......... .......... .......... .......... .......... 50% 38.0M 3s 2024-04-24T16:01:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85700K .......... .......... .......... .......... .......... 50% 23.3M 3s 2024-04-24T16:01:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85750K .......... .......... .......... .......... .......... 50% 30.5M 3s 2024-04-24T16:01:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85800K .......... .......... .......... .......... .......... 50% 29.4M 3s 2024-04-24T16:01:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85850K .......... .......... .......... .......... .......... 50% 45.5M 3s 2024-04-24T16:01:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85900K .......... .......... .......... .......... .......... 50% 32.2M 3s 2024-04-24T16:01:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 85950K .......... .......... .......... .......... .......... 51% 22.1M 3s 2024-04-24T16:01:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86000K .......... .......... .......... .......... .......... 51% 22.5M 3s 2024-04-24T16:01:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86050K .......... .......... .......... .......... .......... 51% 29.2M 3s 2024-04-24T16:01:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86100K .......... .......... .......... .......... .......... 51% 44.3M 3s 2024-04-24T16:01:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86150K .......... .......... .......... .......... .......... 51% 28.5M 3s 2024-04-24T16:01:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86200K .......... .......... .......... .......... .......... 51% 27.8M 3s 2024-04-24T16:01:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86250K .......... .......... .......... .......... .......... 51% 21.9M 3s 2024-04-24T16:01:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86300K .......... .......... .......... .......... .......... 51% 43.0M 3s 2024-04-24T16:01:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86350K .......... .......... .......... .......... .......... 51% 21.9M 3s 2024-04-24T16:01:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86400K .......... .......... .......... .......... .......... 51% 29.9M 3s 2024-04-24T16:01:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86450K .......... .......... .......... .......... .......... 51% 31.3M 3s 2024-04-24T16:01:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86500K .......... .......... .......... .......... .......... 51% 27.3M 3s 2024-04-24T16:01:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86550K .......... .......... .......... .......... .......... 51% 35.9M 3s 2024-04-24T16:01:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86600K .......... .......... .......... .......... .......... 51% 41.2M 3s 2024-04-24T16:01:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86650K .......... .......... .......... .......... .......... 51% 24.3M 3s 2024-04-24T16:01:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86700K .......... .......... .......... .......... .......... 51% 34.4M 3s 2024-04-24T16:01:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86750K .......... .......... .......... .......... .......... 51% 24.3M 3s 2024-04-24T16:01:15.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86800K .......... .......... .......... .......... .......... 51% 21.2M 3s 2024-04-24T16:01:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86850K .......... .......... .......... .......... .......... 51% 28.6M 3s 2024-04-24T16:01:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86900K .......... .......... .......... .......... .......... 51% 56.3M 3s 2024-04-24T16:01:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 86950K .......... .......... .......... .......... .......... 51% 24.1M 3s 2024-04-24T16:01:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87000K .......... .......... .......... .......... .......... 51% 36.1M 3s 2024-04-24T16:01:15.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87050K .......... .......... .......... .......... .......... 51% 28.3M 3s 2024-04-24T16:01:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87100K .......... .......... .......... .......... .......... 51% 28.2M 3s 2024-04-24T16:01:15.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87150K .......... .......... .......... .......... .......... 51% 27.3M 3s 2024-04-24T16:01:15.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87200K .......... .......... .......... .......... .......... 51% 23.7M 3s 2024-04-24T16:01:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87250K .......... .......... .......... .......... .......... 51% 38.7M 3s 2024-04-24T16:01:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87300K .......... .......... .......... .......... .......... 51% 25.0M 3s 2024-04-24T16:01:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87350K .......... .......... .......... .......... .......... 51% 25.1M 3s 2024-04-24T16:01:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87400K .......... .......... .......... .......... .......... 51% 35.3M 3s 2024-04-24T16:01:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87450K .......... .......... .......... .......... .......... 51% 31.2M 3s 2024-04-24T16:01:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87500K .......... .......... .......... .......... .......... 51% 30.0M 3s 2024-04-24T16:01:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87550K .......... .......... .......... .......... .......... 51% 25.8M 3s 2024-04-24T16:01:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87600K .......... .......... .......... .......... .......... 51% 23.6M 3s 2024-04-24T16:01:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87650K .......... .......... .......... .......... .......... 52% 48.8M 3s 2024-04-24T16:01:15.126 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87700K .......... .......... .......... .......... .......... 52% 22.0M 3s 2024-04-24T16:01:15.126 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87750K .......... .......... .......... .......... .......... 52% 47.2M 3s 2024-04-24T16:01:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87800K .......... .......... .......... .......... .......... 52% 28.9M 3s 2024-04-24T16:01:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87850K .......... .......... .......... .......... .......... 52% 37.3M 3s 2024-04-24T16:01:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87900K .......... .......... .......... .......... .......... 52% 24.4M 3s 2024-04-24T16:01:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 87950K .......... .......... .......... .......... .......... 52% 21.5M 3s 2024-04-24T16:01:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88000K .......... .......... .......... .......... .......... 52% 41.2M 3s 2024-04-24T16:01:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88050K .......... .......... .......... .......... .......... 52% 23.1M 3s 2024-04-24T16:01:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88100K .......... .......... .......... .......... .......... 52% 25.4M 3s 2024-04-24T16:01:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88150K .......... .......... .......... .......... .......... 52% 58.3M 3s 2024-04-24T16:01:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88200K .......... .......... .......... .......... .......... 52% 21.6M 3s 2024-04-24T16:01:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88250K .......... .......... .......... .......... .......... 52% 32.2M 3s 2024-04-24T16:01:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88300K .......... .......... .......... .......... .......... 52% 32.2M 3s 2024-04-24T16:01:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88350K .......... .......... .......... .......... .......... 52% 22.4M 3s 2024-04-24T16:01:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88400K .......... .......... .......... .......... .......... 52% 39.5M 3s 2024-04-24T16:01:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88450K .......... .......... .......... .......... .......... 52% 28.3M 3s 2024-04-24T16:01:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88500K .......... .......... .......... .......... .......... 52% 30.0M 3s 2024-04-24T16:01:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88550K .......... .......... .......... .......... .......... 52% 21.5M 3s 2024-04-24T16:01:15.156 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88600K .......... .......... .......... .......... .......... 52% 30.7M 3s 2024-04-24T16:01:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88650K .......... .......... .......... .......... .......... 52% 49.5M 3s 2024-04-24T16:01:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88700K .......... .......... .......... .......... .......... 52% 11.1M 3s 2024-04-24T16:01:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88750K .......... .......... .......... .......... .......... 52% 131M 3s 2024-04-24T16:01:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88800K .......... .......... .......... .......... .......... 52% 64.7M 3s 2024-04-24T16:01:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88850K .......... .......... .......... .......... .......... 52% 34.4M 3s 2024-04-24T16:01:15.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88900K .......... .......... .......... .......... .......... 52% 35.0M 3s 2024-04-24T16:01:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 88950K .......... .......... .......... .......... .......... 52% 24.3M 3s 2024-04-24T16:01:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89000K .......... .......... .......... .......... .......... 52% 37.7M 3s 2024-04-24T16:01:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89050K .......... .......... .......... .......... .......... 52% 21.0M 3s 2024-04-24T16:01:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89100K .......... .......... .......... .......... .......... 52% 23.1M 3s 2024-04-24T16:01:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89150K .......... .......... .......... .......... .......... 52% 30.4M 3s 2024-04-24T16:01:15.175 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89200K .......... .......... .......... .......... .......... 52% 49.1M 3s 2024-04-24T16:01:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89250K .......... .......... .......... .......... .......... 52% 24.2M 3s 2024-04-24T16:01:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89300K .......... .......... .......... .......... .......... 53% 34.5M 3s 2024-04-24T16:01:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89350K .......... .......... .......... .......... .......... 53% 18.4M 3s 2024-04-24T16:01:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89400K .......... .......... .......... .......... .......... 53% 93.7M 3s 2024-04-24T16:01:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89450K .......... .......... .......... .......... .......... 53% 26.6M 3s 2024-04-24T16:01:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89500K .......... .......... .......... .......... .......... 53% 28.5M 3s 2024-04-24T16:01:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89550K .......... .......... .......... .......... .......... 53% 25.5M 3s 2024-04-24T16:01:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89600K .......... .......... .......... .......... .......... 53% 24.4M 3s 2024-04-24T16:01:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89650K .......... .......... .......... .......... .......... 53% 27.7M 3s 2024-04-24T16:01:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89700K .......... .......... .......... .......... .......... 53% 36.9M 3s 2024-04-24T16:01:15.194 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89750K .......... .......... .......... .......... .......... 53% 44.1M 3s 2024-04-24T16:01:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89800K .......... .......... .......... .......... .......... 53% 28.8M 3s 2024-04-24T16:01:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89850K .......... .......... .......... .......... .......... 53% 29.1M 3s 2024-04-24T16:01:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89900K .......... .......... .......... .......... .......... 53% 26.1M 3s 2024-04-24T16:01:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 89950K .......... .......... .......... .......... .......... 53% 26.3M 3s 2024-04-24T16:01:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90000K .......... .......... .......... .......... .......... 53% 21.0M 3s 2024-04-24T16:01:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90050K .......... .......... .......... .......... .......... 53% 50.1M 3s 2024-04-24T16:01:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90100K .......... .......... .......... .......... .......... 53% 22.0M 3s 2024-04-24T16:01:15.208 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90150K .......... .......... .......... .......... .......... 53% 36.3M 3s 2024-04-24T16:01:15.209 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90200K .......... .......... .......... .......... .......... 53% 36.2M 3s 2024-04-24T16:01:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90250K .......... .......... .......... .......... .......... 53% 23.3M 3s 2024-04-24T16:01:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90300K .......... .......... .......... .......... .......... 53% 33.9M 3s 2024-04-24T16:01:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90350K .......... .......... .......... .......... .......... 53% 20.0M 3s 2024-04-24T16:01:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90400K .......... .......... .......... .......... .......... 53% 25.2M 3s 2024-04-24T16:01:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90450K .......... .......... .......... .......... .......... 53% 68.0M 3s 2024-04-24T16:01:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90500K .......... .......... .......... .......... .......... 53% 25.1M 3s 2024-04-24T16:01:15.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90550K .......... .......... .......... .......... .......... 53% 34.6M 3s 2024-04-24T16:01:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90600K .......... .......... .......... .......... .......... 53% 24.9M 3s 2024-04-24T16:01:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90650K .......... .......... .......... .......... .......... 53% 38.2M 3s 2024-04-24T16:01:15.226 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90700K .......... .......... .......... .......... .......... 53% 27.5M 3s 2024-04-24T16:01:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90750K .......... .......... .......... .......... .......... 53% 24.5M 3s 2024-04-24T16:01:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90800K .......... .......... .......... .......... .......... 53% 34.4M 3s 2024-04-24T16:01:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90850K .......... .......... .......... .......... .......... 53% 25.7M 3s 2024-04-24T16:01:15.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90900K .......... .......... .......... .......... .......... 53% 25.0M 3s 2024-04-24T16:01:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 90950K .......... .......... .......... .......... .......... 53% 35.6M 3s 2024-04-24T16:01:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91000K .......... .......... .......... .......... .......... 54% 27.4M 3s 2024-04-24T16:01:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91050K .......... .......... .......... .......... .......... 54% 26.4M 3s 2024-04-24T16:01:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91100K .......... .......... .......... .......... .......... 54% 34.2M 3s 2024-04-24T16:01:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91150K .......... .......... .......... .......... .......... 54% 23.5M 3s 2024-04-24T16:01:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91200K .......... .......... .......... .......... .......... 54% 52.0M 3s 2024-04-24T16:01:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91250K .......... .......... .......... .......... .......... 54% 30.2M 3s 2024-04-24T16:01:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91300K .......... .......... .......... .......... .......... 54% 19.0M 3s 2024-04-24T16:01:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91350K .......... .......... .......... .......... .......... 54% 39.1M 3s 2024-04-24T16:01:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91400K .......... .......... .......... .......... .......... 54% 25.9M 3s 2024-04-24T16:01:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91450K .......... .......... .......... .......... .......... 54% 43.4M 3s 2024-04-24T16:01:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91500K .......... .......... .......... .......... .......... 54% 30.2M 3s 2024-04-24T16:01:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91550K .......... .......... .......... .......... .......... 54% 16.7M 3s 2024-04-24T16:01:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91600K .......... .......... .......... .......... .......... 54% 48.3M 3s 2024-04-24T16:01:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91650K .......... .......... .......... .......... .......... 54% 22.7M 3s 2024-04-24T16:01:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91700K .......... .......... .......... .......... .......... 54% 66.2M 3s 2024-04-24T16:01:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91750K .......... .......... .......... .......... .......... 54% 27.6M 3s 2024-04-24T16:01:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91800K .......... .......... .......... .......... .......... 54% 29.7M 3s 2024-04-24T16:01:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91850K .......... .......... .......... .......... .......... 54% 25.7M 3s 2024-04-24T16:01:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91900K .......... .......... .......... .......... .......... 54% 19.8M 3s 2024-04-24T16:01:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 91950K .......... .......... .......... .......... .......... 54% 41.4M 3s 2024-04-24T16:01:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92000K .......... .......... .......... .......... .......... 54% 38.0M 3s 2024-04-24T16:01:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92050K .......... .......... .......... .......... .......... 54% 26.6M 3s 2024-04-24T16:01:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92100K .......... .......... .......... .......... .......... 54% 25.8M 3s 2024-04-24T16:01:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92150K .......... .......... .......... .......... .......... 54% 31.4M 3s 2024-04-24T16:01:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92200K .......... .......... .......... .......... .......... 54% 44.0M 3s 2024-04-24T16:01:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92250K .......... .......... .......... .......... .......... 54% 15.4M 3s 2024-04-24T16:01:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92300K .......... .......... .......... .......... .......... 54% 54.1M 3s 2024-04-24T16:01:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92350K .......... .......... .......... .......... .......... 54% 26.3M 3s 2024-04-24T16:01:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92400K .......... .......... .......... .......... .......... 54% 28.9M 3s 2024-04-24T16:01:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92450K .......... .......... .......... .......... .......... 54% 28.6M 3s 2024-04-24T16:01:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92500K .......... .......... .......... .......... .......... 54% 49.4M 3s 2024-04-24T16:01:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92550K .......... .......... .......... .......... .......... 54% 20.8M 3s 2024-04-24T16:01:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92600K .......... .......... .......... .......... .......... 54% 45.0M 3s 2024-04-24T16:01:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92650K .......... .......... .......... .......... .......... 54% 23.4M 3s 2024-04-24T16:01:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92700K .......... .......... .......... .......... .......... 55% 44.5M 3s 2024-04-24T16:01:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92750K .......... .......... .......... .......... .......... 55% 26.9M 3s 2024-04-24T16:01:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92800K .......... .......... .......... .......... .......... 55% 21.7M 3s 2024-04-24T16:01:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92850K .......... .......... .......... .......... .......... 55% 24.2M 3s 2024-04-24T16:01:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92900K .......... .......... .......... .......... .......... 55% 31.6M 3s 2024-04-24T16:01:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 92950K .......... .......... .......... .......... .......... 55% 79.9M 3s 2024-04-24T16:01:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93000K .......... .......... .......... .......... .......... 55% 20.7M 3s 2024-04-24T16:01:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93050K .......... .......... .......... .......... .......... 55% 37.0M 3s 2024-04-24T16:01:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93100K .......... .......... .......... .......... .......... 55% 27.1M 3s 2024-04-24T16:01:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93150K .......... .......... .......... .......... .......... 55% 26.0M 3s 2024-04-24T16:01:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93200K .......... .......... .......... .......... .......... 55% 19.1M 3s 2024-04-24T16:01:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93250K .......... .......... .......... .......... .......... 55% 23.6M 3s 2024-04-24T16:01:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93300K .......... .......... .......... .......... .......... 55% 32.5M 3s 2024-04-24T16:01:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93350K .......... .......... .......... .......... .......... 55% 90.9M 3s 2024-04-24T16:01:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93400K .......... .......... .......... .......... .......... 55% 27.7M 3s 2024-04-24T16:01:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93450K .......... .......... .......... .......... .......... 55% 57.5M 3s 2024-04-24T16:01:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93500K .......... .......... .......... .......... .......... 55% 17.6M 3s 2024-04-24T16:01:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93550K .......... .......... .......... .......... .......... 55% 33.4M 3s 2024-04-24T16:01:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93600K .......... .......... .......... .......... .......... 55% 33.0M 3s 2024-04-24T16:01:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93650K .......... .......... .......... .......... .......... 55% 16.7M 3s 2024-04-24T16:01:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93700K .......... .......... .......... .......... .......... 55% 92.4M 3s 2024-04-24T16:01:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93750K .......... .......... .......... .......... .......... 55% 23.3M 3s 2024-04-24T16:01:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93800K .......... .......... .......... .......... .......... 55% 23.7M 3s 2024-04-24T16:01:15.333 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93850K .......... .......... .......... .......... .......... 55% 48.5M 3s 2024-04-24T16:01:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93900K .......... .......... .......... .......... .......... 55% 36.8M 3s 2024-04-24T16:01:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 93950K .......... .......... .......... .......... .......... 55% 15.6M 3s 2024-04-24T16:01:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94000K .......... .......... .......... .......... .......... 55% 115M 3s 2024-04-24T16:01:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94050K .......... .......... .......... .......... .......... 55% 24.6M 3s 2024-04-24T16:01:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94100K .......... .......... .......... .......... .......... 55% 33.4M 3s 2024-04-24T16:01:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94150K .......... .......... .......... .......... .......... 55% 18.1M 3s 2024-04-24T16:01:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94200K .......... .......... .......... .......... .......... 55% 15.0M 3s 2024-04-24T16:01:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94250K .......... .......... .......... .......... .......... 55% 32.5M 3s 2024-04-24T16:01:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94300K .......... .......... .......... .......... .......... 55% 45.6M 3s 2024-04-24T16:01:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94350K .......... .......... .......... .......... .......... 56% 39.4M 3s 2024-04-24T16:01:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94400K .......... .......... .......... .......... .......... 56% 57.8M 3s 2024-04-24T16:01:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94450K .......... .......... .......... .......... .......... 56% 26.2M 3s 2024-04-24T16:01:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94500K .......... .......... .......... .......... .......... 56% 48.5M 3s 2024-04-24T16:01:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94550K .......... .......... .......... .......... .......... 56% 24.1M 3s 2024-04-24T16:01:15.357 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94600K .......... .......... .......... .......... .......... 56% 31.3M 3s 2024-04-24T16:01:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94650K .......... .......... .......... .......... .......... 56% 22.9M 3s 2024-04-24T16:01:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94700K .......... .......... .......... .......... .......... 56% 24.5M 3s 2024-04-24T16:01:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94750K .......... .......... .......... .......... .......... 56% 28.2M 3s 2024-04-24T16:01:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94800K .......... .......... .......... .......... .......... 56% 53.1M 3s 2024-04-24T16:01:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94850K .......... .......... .......... .......... .......... 56% 23.5M 3s 2024-04-24T16:01:15.368 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94900K .......... .......... .......... .......... .......... 56% 37.8M 3s 2024-04-24T16:01:15.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 94950K .......... .......... .......... .......... .......... 56% 39.4M 3s 2024-04-24T16:01:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95000K .......... .......... .......... .......... .......... 56% 28.8M 3s 2024-04-24T16:01:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95050K .......... .......... .......... .......... .......... 56% 24.6M 3s 2024-04-24T16:01:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95100K .......... .......... .......... .......... .......... 56% 26.7M 3s 2024-04-24T16:01:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95150K .......... .......... .......... .......... .......... 56% 20.1M 3s 2024-04-24T16:01:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95200K .......... .......... .......... .......... .......... 56% 39.0M 3s 2024-04-24T16:01:15.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95250K .......... .......... .......... .......... .......... 56% 24.5M 3s 2024-04-24T16:01:15.381 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95300K .......... .......... .......... .......... .......... 56% 49.6M 3s 2024-04-24T16:01:15.384 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95350K .......... .......... .......... .......... .......... 56% 19.7M 3s 2024-04-24T16:01:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95400K .......... .......... .......... .......... .......... 56% 53.2M 3s 2024-04-24T16:01:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95450K .......... .......... .......... .......... .......... 56% 25.6M 3s 2024-04-24T16:01:15.387 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95500K .......... .......... .......... .......... .......... 56% 45.3M 3s 2024-04-24T16:01:15.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95550K .......... .......... .......... .......... .......... 56% 22.3M 3s 2024-04-24T16:01:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95600K .......... .......... .......... .......... .......... 56% 25.2M 3s 2024-04-24T16:01:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95650K .......... .......... .......... .......... .......... 56% 17.4M 3s 2024-04-24T16:01:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95700K .......... .......... .......... .......... .......... 56% 50.5M 3s 2024-04-24T16:01:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95750K .......... .......... .......... .......... .......... 56% 55.2M 3s 2024-04-24T16:01:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95800K .......... .......... .......... .......... .......... 56% 30.5M 3s 2024-04-24T16:01:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95850K .......... .......... .......... .......... .......... 56% 24.0M 3s 2024-04-24T16:01:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95900K .......... .......... .......... .......... .......... 56% 49.8M 3s 2024-04-24T16:01:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 95950K .......... .......... .......... .......... .......... 56% 12.5M 3s 2024-04-24T16:01:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96000K .......... .......... .......... .......... .......... 56% 20.2M 3s 2024-04-24T16:01:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96050K .......... .......... .......... .......... .......... 57% 25.7M 3s 2024-04-24T16:01:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96100K .......... .......... .......... .......... .......... 57% 14.9M 3s 2024-04-24T16:01:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96150K .......... .......... .......... .......... .......... 57% 35.8M 3s 2024-04-24T16:01:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96200K .......... .......... .......... .......... .......... 57% 56.8M 3s 2024-04-24T16:01:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96250K .......... .......... .......... .......... .......... 57% 101M 3s 2024-04-24T16:01:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96300K .......... .......... .......... .......... .......... 57% 24.0M 3s 2024-04-24T16:01:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96350K .......... .......... .......... .......... .......... 57% 32.4M 3s 2024-04-24T16:01:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96400K .......... .......... .......... .......... .......... 57% 61.2M 3s 2024-04-24T16:01:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96450K .......... .......... .......... .......... .......... 57% 29.8M 3s 2024-04-24T16:01:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96500K .......... .......... .......... .......... .......... 57% 43.2M 3s 2024-04-24T16:01:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96550K .......... .......... .......... .......... .......... 57% 52.4M 3s 2024-04-24T16:01:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96600K .......... .......... .......... .......... .......... 57% 24.5M 3s 2024-04-24T16:01:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96650K .......... .......... .......... .......... .......... 57% 45.9M 3s 2024-04-24T16:01:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96700K .......... .......... .......... .......... .......... 57% 20.8M 3s 2024-04-24T16:01:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96750K .......... .......... .......... .......... .......... 57% 24.3M 3s 2024-04-24T16:01:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96800K .......... .......... .......... .......... .......... 57% 45.4M 3s 2024-04-24T16:01:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96850K .......... .......... .......... .......... .......... 57% 26.1M 3s 2024-04-24T16:01:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96900K .......... .......... .......... .......... .......... 57% 35.8M 3s 2024-04-24T16:01:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 96950K .......... .......... .......... .......... .......... 57% 31.4M 3s 2024-04-24T16:01:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97000K .......... .......... .......... .......... .......... 57% 17.8M 3s 2024-04-24T16:01:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97050K .......... .......... .......... .......... .......... 57% 50.3M 3s 2024-04-24T16:01:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97100K .......... .......... .......... .......... .......... 57% 25.0M 3s 2024-04-24T16:01:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97150K .......... .......... .......... .......... .......... 57% 39.4M 3s 2024-04-24T16:01:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97200K .......... .......... .......... .......... .......... 57% 22.7M 3s 2024-04-24T16:01:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97250K .......... .......... .......... .......... .......... 57% 30.2M 3s 2024-04-24T16:01:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97300K .......... .......... .......... .......... .......... 57% 29.8M 3s 2024-04-24T16:01:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97350K .......... .......... .......... .......... .......... 57% 31.0M 3s 2024-04-24T16:01:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97400K .......... .......... .......... .......... .......... 57% 43.6M 3s 2024-04-24T16:01:15.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97450K .......... .......... .......... .......... .......... 57% 24.6M 3s 2024-04-24T16:01:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97500K .......... .......... .......... .......... .......... 57% 33.1M 3s 2024-04-24T16:01:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97550K .......... .......... .......... .......... .......... 57% 17.5M 3s 2024-04-24T16:01:15.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97600K .......... .......... .......... .......... .......... 57% 31.6M 3s 2024-04-24T16:01:15.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97650K .......... .......... .......... .......... .......... 57% 33.6M 3s 2024-04-24T16:01:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97700K .......... .......... .......... .......... .......... 57% 39.9M 3s 2024-04-24T16:01:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97750K .......... .......... .......... .......... .......... 58% 22.8M 3s 2024-04-24T16:01:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97800K .......... .......... .......... .......... .......... 58% 34.3M 3s 2024-04-24T16:01:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97850K .......... .......... .......... .......... .......... 58% 36.4M 3s 2024-04-24T16:01:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97900K .......... .......... .......... .......... .......... 58% 29.4M 3s 2024-04-24T16:01:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 97950K .......... .......... .......... .......... .......... 58% 22.7M 3s 2024-04-24T16:01:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98000K .......... .......... .......... .......... .......... 58% 25.9M 3s 2024-04-24T16:01:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98050K .......... .......... .......... .......... .......... 58% 45.5M 3s 2024-04-24T16:01:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98100K .......... .......... .......... .......... .......... 58% 21.1M 3s 2024-04-24T16:01:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98150K .......... .......... .......... .......... .......... 58% 44.2M 3s 2024-04-24T16:01:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98200K .......... .......... .......... .......... .......... 58% 27.5M 3s 2024-04-24T16:01:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98250K .......... .......... .......... .......... .......... 58% 30.8M 3s 2024-04-24T16:01:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98300K .......... .......... .......... .......... .......... 58% 34.7M 2s 2024-04-24T16:01:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98350K .......... .......... .......... .......... .......... 58% 23.1M 2s 2024-04-24T16:01:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98400K .......... .......... .......... .......... .......... 58% 24.9M 2s 2024-04-24T16:01:15.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98450K .......... .......... .......... .......... .......... 58% 26.4M 2s 2024-04-24T16:01:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98500K .......... .......... .......... .......... .......... 58% 30.8M 2s 2024-04-24T16:01:15.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98550K .......... .......... .......... .......... .......... 58% 47.4M 2s 2024-04-24T16:01:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98600K .......... .......... .......... .......... .......... 58% 21.4M 2s 2024-04-24T16:01:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98650K .......... .......... .......... .......... .......... 58% 47.7M 2s 2024-04-24T16:01:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98700K .......... .......... .......... .......... .......... 58% 24.3M 2s 2024-04-24T16:01:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98750K .......... .......... .......... .......... .......... 58% 23.3M 2s 2024-04-24T16:01:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98800K .......... .......... .......... .......... .......... 58% 20.9M 2s 2024-04-24T16:01:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98850K .......... .......... .......... .......... .......... 58% 78.1M 2s 2024-04-24T16:01:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98900K .......... .......... .......... .......... .......... 58% 31.7M 2s 2024-04-24T16:01:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 98950K .......... .......... .......... .......... .......... 58% 27.2M 2s 2024-04-24T16:01:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99000K .......... .......... .......... .......... .......... 58% 22.9M 2s 2024-04-24T16:01:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99050K .......... .......... .......... .......... .......... 58% 56.2M 2s 2024-04-24T16:01:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99100K .......... .......... .......... .......... .......... 58% 28.7M 2s 2024-04-24T16:01:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99150K .......... .......... .......... .......... .......... 58% 20.6M 2s 2024-04-24T16:01:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99200K .......... .......... .......... .......... .......... 58% 37.6M 2s 2024-04-24T16:01:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99250K .......... .......... .......... .......... .......... 58% 25.2M 2s 2024-04-24T16:01:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99300K .......... .......... .......... .......... .......... 58% 21.8M 2s 2024-04-24T16:01:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99350K .......... .......... .......... .......... .......... 58% 48.1M 2s 2024-04-24T16:01:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99400K .......... .......... .......... .......... .......... 58% 23.8M 2s 2024-04-24T16:01:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99450K .......... .......... .......... .......... .......... 59% 26.5M 2s 2024-04-24T16:01:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99500K .......... .......... .......... .......... .......... 59% 67.4M 2s 2024-04-24T16:01:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99550K .......... .......... .......... .......... .......... 59% 17.7M 2s 2024-04-24T16:01:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99600K .......... .......... .......... .......... .......... 59% 37.0M 2s 2024-04-24T16:01:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99650K .......... .......... .......... .......... .......... 59% 31.9M 2s 2024-04-24T16:01:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99700K .......... .......... .......... .......... .......... 59% 29.3M 2s 2024-04-24T16:01:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99750K .......... .......... .......... .......... .......... 59% 25.7M 2s 2024-04-24T16:01:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99800K .......... .......... .......... .......... .......... 59% 25.9M 2s 2024-04-24T16:01:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99850K .......... .......... .......... .......... .......... 59% 51.8M 2s 2024-04-24T16:01:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99900K .......... .......... .......... .......... .......... 59% 27.8M 2s 2024-04-24T16:01:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 99950K .......... .......... .......... .......... .......... 59% 19.9M 2s 2024-04-24T16:01:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100000K .......... .......... .......... .......... .......... 59% 27.8M 2s 2024-04-24T16:01:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100050K .......... .......... .......... .......... .......... 59% 32.8M 2s 2024-04-24T16:01:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100100K .......... .......... .......... .......... .......... 59% 35.9M 2s 2024-04-24T16:01:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100150K .......... .......... .......... .......... .......... 59% 24.0M 2s 2024-04-24T16:01:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100200K .......... .......... .......... .......... .......... 59% 30.2M 2s 2024-04-24T16:01:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100250K .......... .......... .......... .......... .......... 59% 27.3M 2s 2024-04-24T16:01:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100300K .......... .......... .......... .......... .......... 59% 48.6M 2s 2024-04-24T16:01:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100350K .......... .......... .......... .......... .......... 59% 16.0M 2s 2024-04-24T16:01:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100400K .......... .......... .......... .......... .......... 59% 43.5M 2s 2024-04-24T16:01:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100450K .......... .......... .......... .......... .......... 59% 37.9M 2s 2024-04-24T16:01:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100500K .......... .......... .......... .......... .......... 59% 24.0M 2s 2024-04-24T16:01:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100550K .......... .......... .......... .......... .......... 59% 23.4M 2s 2024-04-24T16:01:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100600K .......... .......... .......... .......... .......... 59% 46.0M 2s 2024-04-24T16:01:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100650K .......... .......... .......... .......... .......... 59% 34.5M 2s 2024-04-24T16:01:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100700K .......... .......... .......... .......... .......... 59% 28.8M 2s 2024-04-24T16:01:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100750K .......... .......... .......... .......... .......... 59% 23.6M 2s 2024-04-24T16:01:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100800K .......... .......... .......... .......... .......... 59% 25.4M 2s 2024-04-24T16:01:15.569 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100850K .......... .......... .......... .......... .......... 59% 29.8M 2s 2024-04-24T16:01:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100900K .......... .......... .......... .......... .......... 59% 31.6M 2s 2024-04-24T16:01:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:100950K .......... .......... .......... .......... .......... 59% 32.2M 2s 2024-04-24T16:01:15.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101000K .......... .......... .......... .......... .......... 59% 24.5M 2s 2024-04-24T16:01:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101050K .......... .......... .......... .......... .......... 59% 26.1M 2s 2024-04-24T16:01:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101100K .......... .......... .......... .......... .......... 60% 26.0M 2s 2024-04-24T16:01:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101150K .......... .......... .......... .......... .......... 60% 27.7M 2s 2024-04-24T16:01:15.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101200K .......... .......... .......... .......... .......... 60% 33.8M 2s 2024-04-24T16:01:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101250K .......... .......... .......... .......... .......... 60% 30.6M 2s 2024-04-24T16:01:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101300K .......... .......... .......... .......... .......... 60% 25.3M 2s 2024-04-24T16:01:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101350K .......... .......... .......... .......... .......... 60% 54.6M 2s 2024-04-24T16:01:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101400K .......... .......... .......... .......... .......... 60% 28.6M 2s 2024-04-24T16:01:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101450K .......... .......... .......... .......... .......... 60% 30.7M 2s 2024-04-24T16:01:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101500K .......... .......... .......... .......... .......... 60% 25.0M 2s 2024-04-24T16:01:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101550K .......... .......... .......... .......... .......... 60% 20.4M 2s 2024-04-24T16:01:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101600K .......... .......... .......... .......... .......... 60% 42.7M 2s 2024-04-24T16:01:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101650K .......... .......... .......... .......... .......... 60% 34.4M 2s 2024-04-24T16:01:15.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101700K .......... .......... .......... .......... .......... 60% 19.8M 2s 2024-04-24T16:01:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101750K .......... .......... .......... .......... .......... 60% 33.6M 2s 2024-04-24T16:01:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101800K .......... .......... .......... .......... .......... 60% 20.3M 2s 2024-04-24T16:01:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101850K .......... .......... .......... .......... .......... 60% 47.7M 2s 2024-04-24T16:01:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101900K .......... .......... .......... .......... .......... 60% 23.9M 2s 2024-04-24T16:01:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:101950K .......... .......... .......... .......... .......... 60% 25.5M 2s 2024-04-24T16:01:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102000K .......... .......... .......... .......... .......... 60% 54.9M 2s 2024-04-24T16:01:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102050K .......... .......... .......... .......... .......... 60% 24.7M 2s 2024-04-24T16:01:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102100K .......... .......... .......... .......... .......... 60% 24.7M 2s 2024-04-24T16:01:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102150K .......... .......... .......... .......... .......... 60% 45.4M 2s 2024-04-24T16:01:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102200K .......... .......... .......... .......... .......... 60% 22.0M 2s 2024-04-24T16:01:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102250K .......... .......... .......... .......... .......... 60% 75.0M 2s 2024-04-24T16:01:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102300K .......... .......... .......... .......... .......... 60% 19.6M 2s 2024-04-24T16:01:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102350K .......... .......... .......... .......... .......... 60% 23.3M 2s 2024-04-24T16:01:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102400K .......... .......... .......... .......... .......... 60% 34.6M 2s 2024-04-24T16:01:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102450K .......... .......... .......... .......... .......... 60% 25.3M 2s 2024-04-24T16:01:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102500K .......... .......... .......... .......... .......... 60% 48.5M 2s 2024-04-24T16:01:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102550K .......... .......... .......... .......... .......... 60% 26.0M 2s 2024-04-24T16:01:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102600K .......... .......... .......... .......... .......... 60% 20.0M 2s 2024-04-24T16:01:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102650K .......... .......... .......... .......... .......... 60% 46.7M 2s 2024-04-24T16:01:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102700K .......... .......... .......... .......... .......... 60% 30.0M 2s 2024-04-24T16:01:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102750K .......... .......... .......... .......... .......... 60% 31.8M 2s 2024-04-24T16:01:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102800K .......... .......... .......... .......... .......... 61% 21.0M 2s 2024-04-24T16:01:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102850K .......... .......... .......... .......... .......... 61% 25.9M 2s 2024-04-24T16:01:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102900K .......... .......... .......... .......... .......... 61% 61.6M 2s 2024-04-24T16:01:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:102950K .......... .......... .......... .......... .......... 61% 25.5M 2s 2024-04-24T16:01:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103000K .......... .......... .......... .......... .......... 61% 34.5M 2s 2024-04-24T16:01:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103050K .......... .......... .......... .......... .......... 61% 21.4M 2s 2024-04-24T16:01:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103100K .......... .......... .......... .......... .......... 61% 27.2M 2s 2024-04-24T16:01:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103150K .......... .......... .......... .......... .......... 61% 21.0M 2s 2024-04-24T16:01:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103200K .......... .......... .......... .......... .......... 61% 83.2M 2s 2024-04-24T16:01:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103250K .......... .......... .......... .......... .......... 61% 34.9M 2s 2024-04-24T16:01:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103300K .......... .......... .......... .......... .......... 61% 21.4M 2s 2024-04-24T16:01:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103350K .......... .......... .......... .......... .......... 61% 26.9M 2s 2024-04-24T16:01:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103400K .......... .......... .......... .......... .......... 61% 62.2M 2s 2024-04-24T16:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103450K .......... .......... .......... .......... .......... 61% 19.8M 2s 2024-04-24T16:01:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103500K .......... .......... .......... .......... .......... 61% 34.1M 2s 2024-04-24T16:01:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103550K .......... .......... .......... .......... .......... 61% 22.9M 2s 2024-04-24T16:01:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103600K .......... .......... .......... .......... .......... 61% 33.6M 2s 2024-04-24T16:01:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103650K .......... .......... .......... .......... .......... 61% 44.3M 2s 2024-04-24T16:01:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103700K .......... .......... .......... .......... .......... 61% 25.0M 2s 2024-04-24T16:01:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103750K .......... .......... .......... .......... .......... 61% 12.4M 2s 2024-04-24T16:01:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103800K .......... .......... .......... .......... .......... 61% 12.8M 2s 2024-04-24T16:01:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103850K .......... .......... .......... .......... .......... 61% 54.5M 2s 2024-04-24T16:01:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103900K .......... .......... .......... .......... .......... 61% 73.8M 2s 2024-04-24T16:01:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:103950K .......... .......... .......... .......... .......... 61% 34.3M 2s 2024-04-24T16:01:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104000K .......... .......... .......... .......... .......... 61% 27.1M 2s 2024-04-24T16:01:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104050K .......... .......... .......... .......... .......... 61% 93.5M 2s 2024-04-24T16:01:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104100K .......... .......... .......... .......... .......... 61% 68.4M 2s 2024-04-24T16:01:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104150K .......... .......... .......... .......... .......... 61% 41.6M 2s 2024-04-24T16:01:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104200K .......... .......... .......... .......... .......... 61% 24.1M 2s 2024-04-24T16:01:15.686 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104250K .......... .......... .......... .......... .......... 61% 18.9M 2s 2024-04-24T16:01:15.687 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104300K .......... .......... .......... .......... .......... 61% 49.9M 2s 2024-04-24T16:01:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104350K .......... .......... .......... .......... .......... 61% 20.7M 2s 2024-04-24T16:01:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104400K .......... .......... .......... .......... .......... 61% 16.2M 2s 2024-04-24T16:01:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104450K .......... .......... .......... .......... .......... 61% 21.3M 2s 2024-04-24T16:01:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104500K .......... .......... .......... .......... .......... 62% 21.8M 2s 2024-04-24T16:01:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104550K .......... .......... .......... .......... .......... 62% 12.6M 2s 2024-04-24T16:01:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104600K .......... .......... .......... .......... .......... 62% 34.5M 2s 2024-04-24T16:01:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104650K .......... .......... .......... .......... .......... 62% 93.1M 2s 2024-04-24T16:01:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104700K .......... .......... .......... .......... .......... 62% 49.4M 2s 2024-04-24T16:01:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104750K .......... .......... .......... .......... .......... 62% 41.0M 2s 2024-04-24T16:01:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104800K .......... .......... .......... .......... .......... 62% 113M 2s 2024-04-24T16:01:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104850K .......... .......... .......... .......... .......... 62% 102M 2s 2024-04-24T16:01:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104900K .......... .......... .......... .......... .......... 62% 28.0M 2s 2024-04-24T16:01:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:104950K .......... .......... .......... .......... .......... 62% 49.5M 2s 2024-04-24T16:01:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105000K .......... .......... .......... .......... .......... 62% 7.57M 2s 2024-04-24T16:01:15.716 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105050K .......... .......... .......... .......... .......... 62% 29.3M 2s 2024-04-24T16:01:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105100K .......... .......... .......... .......... .......... 62% 10.3M 2s 2024-04-24T16:01:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105150K .......... .......... .......... .......... .......... 62% 11.0M 2s 2024-04-24T16:01:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105200K .......... .......... .......... .......... .......... 62% 20.9M 2s 2024-04-24T16:01:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105250K .......... .......... .......... .......... .......... 62% 14.8M 2s 2024-04-24T16:01:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105300K .......... .......... .......... .......... .......... 62% 13.6M 2s 2024-04-24T16:01:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105350K .......... .......... .......... .......... .......... 62% 31.6M 2s 2024-04-24T16:01:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105400K .......... .......... .......... .......... .......... 62% 46.2M 2s 2024-04-24T16:01:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105450K .......... .......... .......... .......... .......... 62% 59.6M 2s 2024-04-24T16:01:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105500K .......... .......... .......... .......... .......... 62% 39.8M 2s 2024-04-24T16:01:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105550K .......... .......... .......... .......... .......... 62% 50.8M 2s 2024-04-24T16:01:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105600K .......... .......... .......... .......... .......... 62% 113M 2s 2024-04-24T16:01:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105650K .......... .......... .......... .......... .......... 62% 17.0M 2s 2024-04-24T16:01:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105700K .......... .......... .......... .......... .......... 62% 33.4M 2s 2024-04-24T16:01:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105750K .......... .......... .......... .......... .......... 62% 9.03M 2s 2024-04-24T16:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105800K .......... .......... .......... .......... .......... 62% 45.3M 2s 2024-04-24T16:01:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105850K .......... .......... .......... .......... .......... 62% 26.0M 2s 2024-04-24T16:01:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105900K .......... .......... .......... .......... .......... 62% 32.0M 2s 2024-04-24T16:01:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:105950K .......... .......... .......... .......... .......... 62% 22.1M 2s 2024-04-24T16:01:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106000K .......... .......... .......... .......... .......... 62% 34.8M 2s 2024-04-24T16:01:15.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106050K .......... .......... .......... .......... .......... 62% 11.2M 2s 2024-04-24T16:01:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106100K .......... .......... .......... .......... .......... 62% 16.6M 2s 2024-04-24T16:01:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106150K .......... .......... .......... .......... .......... 63% 39.2M 2s 2024-04-24T16:01:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106200K .......... .......... .......... .......... .......... 63% 50.4M 2s 2024-04-24T16:01:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106250K .......... .......... .......... .......... .......... 63% 45.8M 2s 2024-04-24T16:01:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106300K .......... .......... .......... .......... .......... 63% 64.2M 2s 2024-04-24T16:01:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106350K .......... .......... .......... .......... .......... 63% 69.4M 2s 2024-04-24T16:01:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106400K .......... .......... .......... .......... .......... 63% 35.1M 2s 2024-04-24T16:01:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106450K .......... .......... .......... .......... .......... 63% 101M 2s 2024-04-24T16:01:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106500K .......... .......... .......... .......... .......... 63% 85.3M 2s 2024-04-24T16:01:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106550K .......... .......... .......... .......... .......... 63% 23.0M 2s 2024-04-24T16:01:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106600K .......... .......... .......... .......... .......... 63% 29.8M 2s 2024-04-24T16:01:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106650K .......... .......... .......... .......... .......... 63% 22.2M 2s 2024-04-24T16:01:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106700K .......... .......... .......... .......... .......... 63% 54.9M 2s 2024-04-24T16:01:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106750K .......... .......... .......... .......... .......... 63% 11.8M 2s 2024-04-24T16:01:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106800K .......... .......... .......... .......... .......... 63% 18.0M 2s 2024-04-24T16:01:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106850K .......... .......... .......... .......... .......... 63% 38.5M 2s 2024-04-24T16:01:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106900K .......... .......... .......... .......... .......... 63% 39.6M 2s 2024-04-24T16:01:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:106950K .......... .......... .......... .......... .......... 63% 98.5M 2s 2024-04-24T16:01:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107000K .......... .......... .......... .......... .......... 63% 36.7M 2s 2024-04-24T16:01:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107050K .......... .......... .......... .......... .......... 63% 50.2M 2s 2024-04-24T16:01:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107100K .......... .......... .......... .......... .......... 63% 48.4M 2s 2024-04-24T16:01:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107150K .......... .......... .......... .......... .......... 63% 43.6M 2s 2024-04-24T16:01:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107200K .......... .......... .......... .......... .......... 63% 51.4M 2s 2024-04-24T16:01:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107250K .......... .......... .......... .......... .......... 63% 110M 2s 2024-04-24T16:01:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107300K .......... .......... .......... .......... .......... 63% 102M 2s 2024-04-24T16:01:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107350K .......... .......... .......... .......... .......... 63% 103M 2s 2024-04-24T16:01:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107400K .......... .......... .......... .......... .......... 63% 31.0M 2s 2024-04-24T16:01:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107450K .......... .......... .......... .......... .......... 63% 49.5M 2s 2024-04-24T16:01:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107500K .......... .......... .......... .......... .......... 63% 42.1M 2s 2024-04-24T16:01:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107550K .......... .......... .......... .......... .......... 63% 52.8M 2s 2024-04-24T16:01:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107600K .......... .......... .......... .......... .......... 63% 49.8M 2s 2024-04-24T16:01:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107650K .......... .......... .......... .......... .......... 63% 43.0M 2s 2024-04-24T16:01:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107700K .......... .......... .......... .......... .......... 63% 58.9M 2s 2024-04-24T16:01:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107750K .......... .......... .......... .......... .......... 63% 49.5M 2s 2024-04-24T16:01:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107800K .......... .......... .......... .......... .......... 63% 48.3M 2s 2024-04-24T16:01:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107850K .......... .......... .......... .......... .......... 64% 41.6M 2s 2024-04-24T16:01:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107900K .......... .......... .......... .......... .......... 64% 21.1M 2s 2024-04-24T16:01:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:107950K .......... .......... .......... .......... .......... 64% 28.1M 2s 2024-04-24T16:01:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108000K .......... .......... .......... .......... .......... 64% 38.1M 2s 2024-04-24T16:01:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108050K .......... .......... .......... .......... .......... 64% 26.7M 2s 2024-04-24T16:01:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108100K .......... .......... .......... .......... .......... 64% 40.9M 2s 2024-04-24T16:01:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108150K .......... .......... .......... .......... .......... 64% 24.4M 2s 2024-04-24T16:01:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108200K .......... .......... .......... .......... .......... 64% 39.0M 2s 2024-04-24T16:01:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108250K .......... .......... .......... .......... .......... 64% 17.0M 2s 2024-04-24T16:01:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108300K .......... .......... .......... .......... .......... 64% 22.3M 2s 2024-04-24T16:01:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108350K .......... .......... .......... .......... .......... 64% 3.82M 2s 2024-04-24T16:01:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108400K .......... .......... .......... .......... .......... 64% 55.7M 2s 2024-04-24T16:01:15.837 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108450K .......... .......... .......... .......... .......... 64% 84.6M 2s 2024-04-24T16:01:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108500K .......... .......... .......... .......... .......... 64% 42.0M 2s 2024-04-24T16:01:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108550K .......... .......... .......... .......... .......... 64% 17.9M 2s 2024-04-24T16:01:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108600K .......... .......... .......... .......... .......... 64% 49.3M 2s 2024-04-24T16:01:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108650K .......... .......... .......... .......... .......... 64% 47.4M 2s 2024-04-24T16:01:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108700K .......... .......... .......... .......... .......... 64% 109M 2s 2024-04-24T16:01:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108750K .......... .......... .......... .......... .......... 64% 17.8M 2s 2024-04-24T16:01:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108800K .......... .......... .......... .......... .......... 64% 103M 2s 2024-04-24T16:01:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108850K .......... .......... .......... .......... .......... 64% 108M 2s 2024-04-24T16:01:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108900K .......... .......... .......... .......... .......... 64% 102M 2s 2024-04-24T16:01:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:108950K .......... .......... .......... .......... .......... 64% 33.3M 2s 2024-04-24T16:01:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109000K .......... .......... .......... .......... .......... 64% 15.4M 2s 2024-04-24T16:01:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109050K .......... .......... .......... .......... .......... 64% 23.1M 2s 2024-04-24T16:01:15.857 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109100K .......... .......... .......... .......... .......... 64% 16.7M 2s 2024-04-24T16:01:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109150K .......... .......... .......... .......... .......... 64% 15.6M 2s 2024-04-24T16:01:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109200K .......... .......... .......... .......... .......... 64% 13.4M 2s 2024-04-24T16:01:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109250K .......... .......... .......... .......... .......... 64% 28.1M 2s 2024-04-24T16:01:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109300K .......... .......... .......... .......... .......... 64% 11.0M 2s 2024-04-24T16:01:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109350K .......... .......... .......... .......... .......... 64% 13.3M 2s 2024-04-24T16:01:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109400K .......... .......... .......... .......... .......... 64% 54.1M 2s 2024-04-24T16:01:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109450K .......... .......... .......... .......... .......... 64% 27.4M 2s 2024-04-24T16:01:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109500K .......... .......... .......... .......... .......... 64% 48.3M 2s 2024-04-24T16:01:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109550K .......... .......... .......... .......... .......... 65% 77.4M 2s 2024-04-24T16:01:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109600K .......... .......... .......... .......... .......... 65% 47.2M 2s 2024-04-24T16:01:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109650K .......... .......... .......... .......... .......... 65% 101M 2s 2024-04-24T16:01:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109700K .......... .......... .......... .......... .......... 65% 82.2M 2s 2024-04-24T16:01:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109750K .......... .......... .......... .......... .......... 65% 61.8M 2s 2024-04-24T16:01:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109800K .......... .......... .......... .......... .......... 65% 22.7M 2s 2024-04-24T16:01:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109850K .......... .......... .......... .......... .......... 65% 36.6M 2s 2024-04-24T16:01:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109900K .......... .......... .......... .......... .......... 65% 22.2M 2s 2024-04-24T16:01:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:109950K .......... .......... .......... .......... .......... 65% 42.0M 2s 2024-04-24T16:01:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110000K .......... .......... .......... .......... .......... 65% 21.8M 2s 2024-04-24T16:01:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110050K .......... .......... .......... .......... .......... 65% 52.6M 2s 2024-04-24T16:01:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110100K .......... .......... .......... .......... .......... 65% 52.4M 2s 2024-04-24T16:01:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110150K .......... .......... .......... .......... .......... 65% 22.7M 2s 2024-04-24T16:01:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110200K .......... .......... .......... .......... .......... 65% 27.3M 2s 2024-04-24T16:01:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110250K .......... .......... .......... .......... .......... 65% 49.5M 2s 2024-04-24T16:01:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110300K .......... .......... .......... .......... .......... 65% 27.2M 2s 2024-04-24T16:01:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110350K .......... .......... .......... .......... .......... 65% 44.1M 2s 2024-04-24T16:01:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110400K .......... .......... .......... .......... .......... 65% 33.8M 2s 2024-04-24T16:01:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110450K .......... .......... .......... .......... .......... 65% 66.3M 2s 2024-04-24T16:01:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110500K .......... .......... .......... .......... .......... 65% 39.6M 2s 2024-04-24T16:01:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110550K .......... .......... .......... .......... .......... 65% 108M 2s 2024-04-24T16:01:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110600K .......... .......... .......... .......... .......... 65% 93.3M 2s 2024-04-24T16:01:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110650K .......... .......... .......... .......... .......... 65% 99.9M 2s 2024-04-24T16:01:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110700K .......... .......... .......... .......... .......... 65% 34.3M 2s 2024-04-24T16:01:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110750K .......... .......... .......... .......... .......... 65% 18.9M 2s 2024-04-24T16:01:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110800K .......... .......... .......... .......... .......... 65% 48.8M 2s 2024-04-24T16:01:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110850K .......... .......... .......... .......... .......... 65% 23.4M 2s 2024-04-24T16:01:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110900K .......... .......... .......... .......... .......... 65% 49.1M 2s 2024-04-24T16:01:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:110950K .......... .......... .......... .......... .......... 65% 22.3M 2s 2024-04-24T16:01:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111000K .......... .......... .......... .......... .......... 65% 55.6M 2s 2024-04-24T16:01:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111050K .......... .......... .......... .......... .......... 65% 23.7M 2s 2024-04-24T16:01:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111100K .......... .......... .......... .......... .......... 65% 32.0M 2s 2024-04-24T16:01:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111150K .......... .......... .......... .......... .......... 65% 29.4M 2s 2024-04-24T16:01:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111200K .......... .......... .......... .......... .......... 65% 23.7M 2s 2024-04-24T16:01:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111250K .......... .......... .......... .......... .......... 66% 20.9M 2s 2024-04-24T16:01:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111300K .......... .......... .......... .......... .......... 66% 57.3M 2s 2024-04-24T16:01:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111350K .......... .......... .......... .......... .......... 66% 31.3M 2s 2024-04-24T16:01:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111400K .......... .......... .......... .......... .......... 66% 58.1M 2s 2024-04-24T16:01:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111450K .......... .......... .......... .......... .......... 66% 81.9M 2s 2024-04-24T16:01:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111500K .......... .......... .......... .......... .......... 66% 64.8M 2s 2024-04-24T16:01:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111550K .......... .......... .......... .......... .......... 66% 17.6M 2s 2024-04-24T16:01:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111600K .......... .......... .......... .......... .......... 66% 23.0M 2s 2024-04-24T16:01:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111650K .......... .......... .......... .......... .......... 66% 22.7M 2s 2024-04-24T16:01:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111700K .......... .......... .......... .......... .......... 66% 53.3M 2s 2024-04-24T16:01:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111750K .......... .......... .......... .......... .......... 66% 16.7M 2s 2024-04-24T16:01:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111800K .......... .......... .......... .......... .......... 66% 32.6M 2s 2024-04-24T16:01:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111850K .......... .......... .......... .......... .......... 66% 31.6M 2s 2024-04-24T16:01:15.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111900K .......... .......... .......... .......... .......... 66% 50.2M 2s 2024-04-24T16:01:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:111950K .......... .......... .......... .......... .......... 66% 17.5M 2s 2024-04-24T16:01:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112000K .......... .......... .......... .......... .......... 66% 22.7M 2s 2024-04-24T16:01:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112050K .......... .......... .......... .......... .......... 66% 32.7M 2s 2024-04-24T16:01:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112100K .......... .......... .......... .......... .......... 66% 48.7M 2s 2024-04-24T16:01:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112150K .......... .......... .......... .......... .......... 66% 68.9M 2s 2024-04-24T16:01:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112200K .......... .......... .......... .......... .......... 66% 24.4M 2s 2024-04-24T16:01:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112250K .......... .......... .......... .......... .......... 66% 22.0M 2s 2024-04-24T16:01:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112300K .......... .......... .......... .......... .......... 66% 48.4M 2s 2024-04-24T16:01:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112350K .......... .......... .......... .......... .......... 66% 20.7M 2s 2024-04-24T16:01:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112400K .......... .......... .......... .......... .......... 66% 30.9M 2s 2024-04-24T16:01:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112450K .......... .......... .......... .......... .......... 66% 39.1M 2s 2024-04-24T16:01:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112500K .......... .......... .......... .......... .......... 66% 23.3M 2s 2024-04-24T16:01:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112550K .......... .......... .......... .......... .......... 66% 29.0M 2s 2024-04-24T16:01:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112600K .......... .......... .......... .......... .......... 66% 22.1M 2s 2024-04-24T16:01:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112650K .......... .......... .......... .......... .......... 66% 30.0M 2s 2024-04-24T16:01:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112700K .......... .......... .......... .......... .......... 66% 22.6M 2s 2024-04-24T16:01:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112750K .......... .......... .......... .......... .......... 66% 16.9M 2s 2024-04-24T16:01:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112800K .......... .......... .......... .......... .......... 66% 29.8M 2s 2024-04-24T16:01:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112850K .......... .......... .......... .......... .......... 66% 46.2M 2s 2024-04-24T16:01:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112900K .......... .......... .......... .......... .......... 67% 20.7M 2s 2024-04-24T16:01:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:112950K .......... .......... .......... .......... .......... 67% 44.4M 2s 2024-04-24T16:01:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113000K .......... .......... .......... .......... .......... 67% 19.9M 2s 2024-04-24T16:01:15.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113050K .......... .......... .......... .......... .......... 67% 88.9M 2s 2024-04-24T16:01:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113100K .......... .......... .......... .......... .......... 67% 31.0M 2s 2024-04-24T16:01:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113150K .......... .......... .......... .......... .......... 67% 22.0M 2s 2024-04-24T16:01:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113200K .......... .......... .......... .......... .......... 67% 31.6M 2s 2024-04-24T16:01:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113250K .......... .......... .......... .......... .......... 67% 31.6M 2s 2024-04-24T16:01:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113300K .......... .......... .......... .......... .......... 67% 31.7M 2s 2024-04-24T16:01:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113350K .......... .......... .......... .......... .......... 67% 22.9M 2s 2024-04-24T16:01:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113400K .......... .......... .......... .......... .......... 67% 32.3M 2s 2024-04-24T16:01:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113450K .......... .......... .......... .......... .......... 67% 43.9M 2s 2024-04-24T16:01:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113500K .......... .......... .......... .......... .......... 67% 33.9M 2s 2024-04-24T16:01:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113550K .......... .......... .......... .......... .......... 67% 22.1M 2s 2024-04-24T16:01:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113600K .......... .......... .......... .......... .......... 67% 31.5M 2s 2024-04-24T16:01:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113650K .......... .......... .......... .......... .......... 67% 31.3M 2s 2024-04-24T16:01:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113700K .......... .......... .......... .......... .......... 67% 31.4M 2s 2024-04-24T16:01:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113750K .......... .......... .......... .......... .......... 67% 20.0M 2s 2024-04-24T16:01:16.014 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113800K .......... .......... .......... .......... .......... 67% 50.0M 2s 2024-04-24T16:01:16.019 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113850K .......... .......... .......... .......... .......... 67% 45.0M 2s 2024-04-24T16:01:16.019 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113900K .......... .......... .......... .......... .......... 67% 118M 2s 2024-04-24T16:01:16.019 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:113950K .......... .......... .......... .......... .......... 67% 28.3M 2s 2024-04-24T16:01:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114000K .......... .......... .......... .......... .......... 67% 23.7M 2s 2024-04-24T16:01:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114050K .......... .......... .......... .......... .......... 67% 47.9M 2s 2024-04-24T16:01:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114100K .......... .......... .......... .......... .......... 67% 29.5M 2s 2024-04-24T16:01:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114150K .......... .......... .......... .......... .......... 67% 28.1M 2s 2024-04-24T16:01:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114200K .......... .......... .......... .......... .......... 67% 25.3M 2s 2024-04-24T16:01:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114250K .......... .......... .......... .......... .......... 67% 40.8M 2s 2024-04-24T16:01:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114300K .......... .......... .......... .......... .......... 67% 30.1M 2s 2024-04-24T16:01:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114350K .......... .......... .......... .......... .......... 67% 23.4M 2s 2024-04-24T16:01:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114400K .......... .......... .......... .......... .......... 67% 24.0M 2s 2024-04-24T16:01:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114450K .......... .......... .......... .......... .......... 67% 31.7M 2s 2024-04-24T16:01:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114500K .......... .......... .......... .......... .......... 67% 25.1M 2s 2024-04-24T16:01:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114550K .......... .......... .......... .......... .......... 67% 37.7M 2s 2024-04-24T16:01:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114600K .......... .......... .......... .......... .......... 68% 30.8M 2s 2024-04-24T16:01:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114650K .......... .......... .......... .......... .......... 68% 25.2M 2s 2024-04-24T16:01:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114700K .......... .......... .......... .......... .......... 68% 32.1M 2s 2024-04-24T16:01:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114750K .......... .......... .......... .......... .......... 68% 22.6M 2s 2024-04-24T16:01:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114800K .......... .......... .......... .......... .......... 68% 24.9M 2s 2024-04-24T16:01:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114850K .......... .......... .......... .......... .......... 68% 51.7M 2s 2024-04-24T16:01:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114900K .......... .......... .......... .......... .......... 68% 30.3M 2s 2024-04-24T16:01:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:114950K .......... .......... .......... .......... .......... 68% 19.2M 2s 2024-04-24T16:01:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115000K .......... .......... .......... .......... .......... 68% 69.3M 2s 2024-04-24T16:01:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115050K .......... .......... .......... .......... .......... 68% 34.1M 2s 2024-04-24T16:01:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115100K .......... .......... .......... .......... .......... 68% 23.9M 2s 2024-04-24T16:01:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115150K .......... .......... .......... .......... .......... 68% 22.0M 2s 2024-04-24T16:01:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115200K .......... .......... .......... .......... .......... 68% 29.4M 2s 2024-04-24T16:01:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115250K .......... .......... .......... .......... .......... 68% 31.0M 2s 2024-04-24T16:01:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115300K .......... .......... .......... .......... .......... 68% 29.0M 2s 2024-04-24T16:01:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115350K .......... .......... .......... .......... .......... 68% 34.7M 2s 2024-04-24T16:01:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115400K .......... .......... .......... .......... .......... 68% 24.1M 2s 2024-04-24T16:01:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115450K .......... .......... .......... .......... .......... 68% 27.4M 2s 2024-04-24T16:01:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115500K .......... .......... .......... .......... .......... 68% 34.7M 2s 2024-04-24T16:01:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115550K .......... .......... .......... .......... .......... 68% 23.4M 2s 2024-04-24T16:01:16.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115600K .......... .......... .......... .......... .......... 68% 44.4M 2s 2024-04-24T16:01:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115650K .......... .......... .......... .......... .......... 68% 25.4M 2s 2024-04-24T16:01:16.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115700K .......... .......... .......... .......... .......... 68% 22.6M 2s 2024-04-24T16:01:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115750K .......... .......... .......... .......... .......... 68% 39.3M 2s 2024-04-24T16:01:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115800K .......... .......... .......... .......... .......... 68% 25.1M 2s 2024-04-24T16:01:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115850K .......... .......... .......... .......... .......... 68% 47.1M 2s 2024-04-24T16:01:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115900K .......... .......... .......... .......... .......... 68% 23.9M 2s 2024-04-24T16:01:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:115950K .......... .......... .......... .......... .......... 68% 27.7M 2s 2024-04-24T16:01:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116000K .......... .......... .......... .......... .......... 68% 22.1M 2s 2024-04-24T16:01:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116050K .......... .......... .......... .......... .......... 68% 46.7M 2s 2024-04-24T16:01:16.086 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116100K .......... .......... .......... .......... .......... 68% 36.6M 2s 2024-04-24T16:01:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116150K .......... .......... .......... .......... .......... 68% 24.0M 2s 2024-04-24T16:01:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116200K .......... .......... .......... .......... .......... 68% 37.5M 2s 2024-04-24T16:01:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116250K .......... .......... .......... .......... .......... 68% 27.9M 2s 2024-04-24T16:01:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116300K .......... .......... .......... .......... .......... 69% 24.5M 2s 2024-04-24T16:01:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116350K .......... .......... .......... .......... .......... 69% 21.1M 2s 2024-04-24T16:01:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116400K .......... .......... .......... .......... .......... 69% 51.4M 2s 2024-04-24T16:01:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116450K .......... .......... .......... .......... .......... 69% 23.5M 2s 2024-04-24T16:01:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116500K .......... .......... .......... .......... .......... 69% 36.8M 2s 2024-04-24T16:01:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116550K .......... .......... .......... .......... .......... 69% 25.8M 2s 2024-04-24T16:01:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116600K .......... .......... .......... .......... .......... 69% 49.5M 2s 2024-04-24T16:01:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116650K .......... .......... .......... .......... .......... 69% 22.5M 2s 2024-04-24T16:01:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116700K .......... .......... .......... .......... .......... 69% 42.2M 2s 2024-04-24T16:01:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116750K .......... .......... .......... .......... .......... 69% 21.1M 2s 2024-04-24T16:01:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116800K .......... .......... .......... .......... .......... 69% 24.3M 2s 2024-04-24T16:01:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116850K .......... .......... .......... .......... .......... 69% 26.8M 2s 2024-04-24T16:01:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116900K .......... .......... .......... .......... .......... 69% 43.9M 2s 2024-04-24T16:01:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:116950K .......... .......... .......... .......... .......... 69% 28.5M 2s 2024-04-24T16:01:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117000K .......... .......... .......... .......... .......... 69% 31.4M 2s 2024-04-24T16:01:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117050K .......... .......... .......... .......... .......... 69% 24.1M 2s 2024-04-24T16:01:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117100K .......... .......... .......... .......... .......... 69% 41.9M 2s 2024-04-24T16:01:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117150K .......... .......... .......... .......... .......... 69% 23.2M 2s 2024-04-24T16:01:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117200K .......... .......... .......... .......... .......... 69% 25.3M 2s 2024-04-24T16:01:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117250K .......... .......... .......... .......... .......... 69% 36.7M 2s 2024-04-24T16:01:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117300K .......... .......... .......... .......... .......... 69% 24.6M 2s 2024-04-24T16:01:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117350K .......... .......... .......... .......... .......... 69% 24.5M 2s 2024-04-24T16:01:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117400K .......... .......... .......... .......... .......... 69% 75.4M 2s 2024-04-24T16:01:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117450K .......... .......... .......... .......... .......... 69% 20.8M 2s 2024-04-24T16:01:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117500K .......... .......... .......... .......... .......... 69% 35.6M 2s 2024-04-24T16:01:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117550K .......... .......... .......... .......... .......... 69% 23.6M 2s 2024-04-24T16:01:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117600K .......... .......... .......... .......... .......... 69% 24.9M 2s 2024-04-24T16:01:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117650K .......... .......... .......... .......... .......... 69% 49.5M 2s 2024-04-24T16:01:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117700K .......... .......... .......... .......... .......... 69% 24.5M 2s 2024-04-24T16:01:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117750K .......... .......... .......... .......... .......... 69% 34.5M 2s 2024-04-24T16:01:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117800K .......... .......... .......... .......... .......... 69% 23.7M 2s 2024-04-24T16:01:16.145 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117850K .......... .......... .......... .......... .......... 69% 45.8M 2s 2024-04-24T16:01:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117900K .......... .......... .......... .......... .......... 69% 21.7M 2s 2024-04-24T16:01:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:117950K .......... .......... .......... .......... .......... 70% 25.9M 2s 2024-04-24T16:01:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118000K .......... .......... .......... .......... .......... 70% 32.6M 2s 2024-04-24T16:01:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118050K .......... .......... .......... .......... .......... 70% 27.9M 2s 2024-04-24T16:01:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118100K .......... .......... .......... .......... .......... 70% 22.3M 2s 2024-04-24T16:01:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118150K .......... .......... .......... .......... .......... 70% 50.9M 2s 2024-04-24T16:01:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118200K .......... .......... .......... .......... .......... 70% 12.8M 2s 2024-04-24T16:01:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118250K .......... .......... .......... .......... .......... 70% 267M 2s 2024-04-24T16:01:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118300K .......... .......... .......... .......... .......... 70% 34.3M 2s 2024-04-24T16:01:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118350K .......... .......... .......... .......... .......... 70% 23.6M 2s 2024-04-24T16:01:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118400K .......... .......... .......... .......... .......... 70% 46.8M 2s 2024-04-24T16:01:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118450K .......... .......... .......... .......... .......... 70% 25.3M 2s 2024-04-24T16:01:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118500K .......... .......... .......... .......... .......... 70% 29.6M 2s 2024-04-24T16:01:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118550K .......... .......... .......... .......... .......... 70% 24.9M 2s 2024-04-24T16:01:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118600K .......... .......... .......... .......... .......... 70% 37.4M 2s 2024-04-24T16:01:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118650K .......... .......... .......... .......... .......... 70% 32.5M 2s 2024-04-24T16:01:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118700K .......... .......... .......... .......... .......... 70% 28.4M 2s 2024-04-24T16:01:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118750K .......... .......... .......... .......... .......... 70% 19.9M 2s 2024-04-24T16:01:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118800K .......... .......... .......... .......... .......... 70% 35.3M 2s 2024-04-24T16:01:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118850K .......... .......... .......... .......... .......... 70% 24.7M 2s 2024-04-24T16:01:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118900K .......... .......... .......... .......... .......... 70% 51.8M 2s 2024-04-24T16:01:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:118950K .......... .......... .......... .......... .......... 70% 26.6M 2s 2024-04-24T16:01:16.184 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119000K .......... .......... .......... .......... .......... 70% 25.0M 2s 2024-04-24T16:01:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119050K .......... .......... .......... .......... .......... 70% 31.2M 2s 2024-04-24T16:01:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119100K .......... .......... .......... .......... .......... 70% 21.4M 2s 2024-04-24T16:01:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119150K .......... .......... .......... .......... .......... 70% 24.2M 2s 2024-04-24T16:01:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119200K .......... .......... .......... .......... .......... 70% 45.4M 2s 2024-04-24T16:01:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119250K .......... .......... .......... .......... .......... 70% 37.4M 2s 2024-04-24T16:01:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119300K .......... .......... .......... .......... .......... 70% 24.0M 2s 2024-04-24T16:01:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119350K .......... .......... .......... .......... .......... 70% 29.8M 2s 2024-04-24T16:01:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119400K .......... .......... .......... .......... .......... 70% 27.6M 2s 2024-04-24T16:01:16.201 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119450K .......... .......... .......... .......... .......... 70% 20.7M 2s 2024-04-24T16:01:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119500K .......... .......... .......... .......... .......... 70% 25.0M 2s 2024-04-24T16:01:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119550K .......... .......... .......... .......... .......... 70% 31.2M 2s 2024-04-24T16:01:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119600K .......... .......... .......... .......... .......... 70% 43.5M 2s 2024-04-24T16:01:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119650K .......... .......... .......... .......... .......... 71% 24.5M 2s 2024-04-24T16:01:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119700K .......... .......... .......... .......... .......... 71% 52.6M 2s 2024-04-24T16:01:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119750K .......... .......... .......... .......... .......... 71% 30.8M 2s 2024-04-24T16:01:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119800K .......... .......... .......... .......... .......... 71% 25.8M 2s 2024-04-24T16:01:16.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119850K .......... .......... .......... .......... .......... 71% 25.3M 2s 2024-04-24T16:01:16.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119900K .......... .......... .......... .......... .......... 71% 54.0M 2s 2024-04-24T16:01:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:119950K .......... .......... .......... .......... .......... 71% 23.4M 2s 2024-04-24T16:01:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120000K .......... .......... .......... .......... .......... 71% 30.2M 2s 2024-04-24T16:01:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120050K .......... .......... .......... .......... .......... 71% 26.3M 2s 2024-04-24T16:01:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120100K .......... .......... .......... .......... .......... 71% 24.4M 2s 2024-04-24T16:01:16.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120150K .......... .......... .......... .......... .......... 71% 24.1M 2s 2024-04-24T16:01:16.225 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120200K .......... .......... .......... .......... .......... 71% 97.4M 2s 2024-04-24T16:01:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120250K .......... .......... .......... .......... .......... 71% 23.3M 2s 2024-04-24T16:01:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120300K .......... .......... .......... .......... .......... 71% 28.7M 2s 2024-04-24T16:01:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120350K .......... .......... .......... .......... .......... 71% 18.3M 2s 2024-04-24T16:01:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120400K .......... .......... .......... .......... .......... 71% 18.3M 2s 2024-04-24T16:01:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120450K .......... .......... .......... .......... .......... 71% 32.7M 2s 2024-04-24T16:01:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120500K .......... .......... .......... .......... .......... 71% 22.3M 2s 2024-04-24T16:01:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120550K .......... .......... .......... .......... .......... 71% 18.7M 2s 2024-04-24T16:01:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120600K .......... .......... .......... .......... .......... 71% 19.9M 2s 2024-04-24T16:01:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120650K .......... .......... .......... .......... .......... 71% 34.7M 2s 2024-04-24T16:01:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120700K .......... .......... .......... .......... .......... 71% 20.6M 2s 2024-04-24T16:01:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120750K .......... .......... .......... .......... .......... 71% 19.0M 2s 2024-04-24T16:01:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120800K .......... .......... .......... .......... .......... 71% 113M 2s 2024-04-24T16:01:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120850K .......... .......... .......... .......... .......... 71% 83.5M 2s 2024-04-24T16:01:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120900K .......... .......... .......... .......... .......... 71% 99.5M 2s 2024-04-24T16:01:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:120950K .......... .......... .......... .......... .......... 71% 103M 2s 2024-04-24T16:01:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121000K .......... .......... .......... .......... .......... 71% 35.2M 2s 2024-04-24T16:01:16.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121050K .......... .......... .......... .......... .......... 71% 25.1M 2s 2024-04-24T16:01:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121100K .......... .......... .......... .......... .......... 71% 29.7M 2s 2024-04-24T16:01:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121150K .......... .......... .......... .......... .......... 71% 23.5M 2s 2024-04-24T16:01:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121200K .......... .......... .......... .......... .......... 71% 46.0M 2s 2024-04-24T16:01:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121250K .......... .......... .......... .......... .......... 71% 24.8M 2s 2024-04-24T16:01:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121300K .......... .......... .......... .......... .......... 71% 35.7M 2s 2024-04-24T16:01:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121350K .......... .......... .......... .......... .......... 72% 22.0M 2s 2024-04-24T16:01:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121400K .......... .......... .......... .......... .......... 72% 20.9M 2s 2024-04-24T16:01:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121450K .......... .......... .......... .......... .......... 72% 42.4M 2s 2024-04-24T16:01:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121500K .......... .......... .......... .......... .......... 72% 18.6M 2s 2024-04-24T16:01:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121550K .......... .......... .......... .......... .......... 72% 14.1M 2s 2024-04-24T16:01:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121600K .......... .......... .......... .......... .......... 72% 25.9M 2s 2024-04-24T16:01:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121650K .......... .......... .......... .......... .......... 72% 23.0M 2s 2024-04-24T16:01:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121700K .......... .......... .......... .......... .......... 72% 45.5M 2s 2024-04-24T16:01:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121750K .......... .......... .......... .......... .......... 72% 21.7M 2s 2024-04-24T16:01:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121800K .......... .......... .......... .......... .......... 72% 28.0M 2s 2024-04-24T16:01:16.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121850K .......... .......... .......... .......... .......... 72% 19.6M 2s 2024-04-24T16:01:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121900K .......... .......... .......... .......... .......... 72% 23.0M 2s 2024-04-24T16:01:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:121950K .......... .......... .......... .......... .......... 72% 17.4M 2s 2024-04-24T16:01:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122000K .......... .......... .......... .......... .......... 72% 37.0M 2s 2024-04-24T16:01:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122050K .......... .......... .......... .......... .......... 72% 20.6M 2s 2024-04-24T16:01:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122100K .......... .......... .......... .......... .......... 72% 112M 2s 2024-04-24T16:01:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122150K .......... .......... .......... .......... .......... 72% 96.0M 2s 2024-04-24T16:01:16.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122200K .......... .......... .......... .......... .......... 72% 110M 2s 2024-04-24T16:01:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122250K .......... .......... .......... .......... .......... 72% 91.2M 2s 2024-04-24T16:01:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122300K .......... .......... .......... .......... .......... 72% 46.8M 2s 2024-04-24T16:01:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122350K .......... .......... .......... .......... .......... 72% 22.4M 2s 2024-04-24T16:01:16.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122400K .......... .......... .......... .......... .......... 72% 22.5M 2s 2024-04-24T16:01:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122450K .......... .......... .......... .......... .......... 72% 23.9M 2s 2024-04-24T16:01:16.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122500K .......... .......... .......... .......... .......... 72% 37.3M 2s 2024-04-24T16:01:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122550K .......... .......... .......... .......... .......... 72% 12.7M 2s 2024-04-24T16:01:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122600K .......... .......... .......... .......... .......... 72% 32.9M 2s 2024-04-24T16:01:16.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122650K .......... .......... .......... .......... .......... 72% 22.2M 2s 2024-04-24T16:01:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122700K .......... .......... .......... .......... .......... 72% 47.0M 2s 2024-04-24T16:01:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122750K .......... .......... .......... .......... .......... 72% 24.7M 2s 2024-04-24T16:01:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122800K .......... .......... .......... .......... .......... 72% 22.0M 2s 2024-04-24T16:01:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122850K .......... .......... .......... .......... .......... 72% 31.3M 2s 2024-04-24T16:01:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122900K .......... .......... .......... .......... .......... 72% 37.6M 2s 2024-04-24T16:01:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:122950K .......... .......... .......... .......... .......... 72% 27.0M 2s 2024-04-24T16:01:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123000K .......... .......... .......... .......... .......... 72% 27.7M 2s 2024-04-24T16:01:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123050K .......... .......... .......... .......... .......... 73% 33.8M 2s 2024-04-24T16:01:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123100K .......... .......... .......... .......... .......... 73% 17.5M 2s 2024-04-24T16:01:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123150K .......... .......... .......... .......... .......... 73% 37.5M 2s 2024-04-24T16:01:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123200K .......... .......... .......... .......... .......... 73% 25.1M 2s 2024-04-24T16:01:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123250K .......... .......... .......... .......... .......... 73% 50.2M 2s 2024-04-24T16:01:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123300K .......... .......... .......... .......... .......... 73% 41.2M 2s 2024-04-24T16:01:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123350K .......... .......... .......... .......... .......... 73% 34.5M 2s 2024-04-24T16:01:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123400K .......... .......... .......... .......... .......... 73% 49.6M 2s 2024-04-24T16:01:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123450K .......... .......... .......... .......... .......... 73% 50.5M 2s 2024-04-24T16:01:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123500K .......... .......... .......... .......... .......... 73% 28.6M 2s 2024-04-24T16:01:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123550K .......... .......... .......... .......... .......... 73% 25.5M 2s 2024-04-24T16:01:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123600K .......... .......... .......... .......... .......... 73% 26.9M 2s 2024-04-24T16:01:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123650K .......... .......... .......... .......... .......... 73% 36.2M 2s 2024-04-24T16:01:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123700K .......... .......... .......... .......... .......... 73% 29.0M 2s 2024-04-24T16:01:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123750K .......... .......... .......... .......... .......... 73% 22.5M 2s 2024-04-24T16:01:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123800K .......... .......... .......... .......... .......... 73% 43.0M 2s 2024-04-24T16:01:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123850K .......... .......... .......... .......... .......... 73% 27.1M 2s 2024-04-24T16:01:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123900K .......... .......... .......... .......... .......... 73% 31.2M 2s 2024-04-24T16:01:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:123950K .......... .......... .......... .......... .......... 73% 18.2M 2s 2024-04-24T16:01:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124000K .......... .......... .......... .......... .......... 73% 25.7M 2s 2024-04-24T16:01:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124050K .......... .......... .......... .......... .......... 73% 21.2M 2s 2024-04-24T16:01:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124100K .......... .......... .......... .......... .......... 73% 35.6M 2s 2024-04-24T16:01:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124150K .......... .......... .......... .......... .......... 73% 18.0M 2s 2024-04-24T16:01:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124200K .......... .......... .......... .......... .......... 73% 43.4M 2s 2024-04-24T16:01:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124250K .......... .......... .......... .......... .......... 73% 28.3M 2s 2024-04-24T16:01:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124300K .......... .......... .......... .......... .......... 73% 32.3M 2s 2024-04-24T16:01:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124350K .......... .......... .......... .......... .......... 73% 33.3M 2s 2024-04-24T16:01:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124400K .......... .......... .......... .......... .......... 73% 22.8M 2s 2024-04-24T16:01:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124450K .......... .......... .......... .......... .......... 73% 28.8M 2s 2024-04-24T16:01:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124500K .......... .......... .......... .......... .......... 73% 49.0M 2s 2024-04-24T16:01:16.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124550K .......... .......... .......... .......... .......... 73% 24.7M 2s 2024-04-24T16:01:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124600K .......... .......... .......... .......... .......... 73% 37.4M 2s 2024-04-24T16:01:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124650K .......... .......... .......... .......... .......... 73% 19.1M 2s 2024-04-24T16:01:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124700K .......... .......... .......... .......... .......... 74% 28.6M 2s 2024-04-24T16:01:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124750K .......... .......... .......... .......... .......... 74% 21.0M 2s 2024-04-24T16:01:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124800K .......... .......... .......... .......... .......... 74% 40.2M 2s 2024-04-24T16:01:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124850K .......... .......... .......... .......... .......... 74% 42.8M 2s 2024-04-24T16:01:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124900K .......... .......... .......... .......... .......... 74% 9.40M 2s 2024-04-24T16:01:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:124950K .......... .......... .......... .......... .......... 74% 21.8M 2s 2024-04-24T16:01:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125000K .......... .......... .......... .......... .......... 74% 43.5M 2s 2024-04-24T16:01:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125050K .......... .......... .......... .......... .......... 74% 21.4M 2s 2024-04-24T16:01:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125100K .......... .......... .......... .......... .......... 74% 32.3M 2s 2024-04-24T16:01:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125150K .......... .......... .......... .......... .......... 74% 18.6M 2s 2024-04-24T16:01:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125200K .......... .......... .......... .......... .......... 74% 32.8M 2s 2024-04-24T16:01:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125250K .......... .......... .......... .......... .......... 74% 20.0M 2s 2024-04-24T16:01:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125300K .......... .......... .......... .......... .......... 74% 42.3M 2s 2024-04-24T16:01:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125350K .......... .......... .......... .......... .......... 74% 27.2M 2s 2024-04-24T16:01:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125400K .......... .......... .......... .......... .......... 74% 49.6M 2s 2024-04-24T16:01:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125450K .......... .......... .......... .......... .......... 74% 26.6M 2s 2024-04-24T16:01:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125500K .......... .......... .......... .......... .......... 74% 43.6M 2s 2024-04-24T16:01:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125550K .......... .......... .......... .......... .......... 74% 28.6M 2s 2024-04-24T16:01:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125600K .......... .......... .......... .......... .......... 74% 27.7M 2s 2024-04-24T16:01:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125650K .......... .......... .......... .......... .......... 74% 48.8M 2s 2024-04-24T16:01:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125700K .......... .......... .......... .......... .......... 74% 26.4M 2s 2024-04-24T16:01:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125750K .......... .......... .......... .......... .......... 74% 42.0M 2s 2024-04-24T16:01:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125800K .......... .......... .......... .......... .......... 74% 27.6M 2s 2024-04-24T16:01:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125850K .......... .......... .......... .......... .......... 74% 33.1M 2s 2024-04-24T16:01:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125900K .......... .......... .......... .......... .......... 74% 27.9M 2s 2024-04-24T16:01:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:125950K .......... .......... .......... .......... .......... 74% 26.9M 2s 2024-04-24T16:01:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126000K .......... .......... .......... .......... .......... 74% 34.9M 1s 2024-04-24T16:01:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126050K .......... .......... .......... .......... .......... 74% 47.3M 1s 2024-04-24T16:01:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126100K .......... .......... .......... .......... .......... 74% 24.9M 1s 2024-04-24T16:01:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126150K .......... .......... .......... .......... .......... 74% 54.2M 1s 2024-04-24T16:01:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126200K .......... .......... .......... .......... .......... 74% 25.5M 1s 2024-04-24T16:01:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126250K .......... .......... .......... .......... .......... 74% 35.7M 1s 2024-04-24T16:01:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126300K .......... .......... .......... .......... .......... 74% 25.4M 1s 2024-04-24T16:01:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126350K .......... .......... .......... .......... .......... 74% 25.6M 1s 2024-04-24T16:01:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126400K .......... .......... .......... .......... .......... 75% 26.5M 1s 2024-04-24T16:01:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126450K .......... .......... .......... .......... .......... 75% 26.3M 1s 2024-04-24T16:01:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126500K .......... .......... .......... .......... .......... 75% 20.4M 1s 2024-04-24T16:01:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126550K .......... .......... .......... .......... .......... 75% 43.4M 1s 2024-04-24T16:01:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126600K .......... .......... .......... .......... .......... 75% 29.8M 1s 2024-04-24T16:01:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126650K .......... .......... .......... .......... .......... 75% 29.7M 1s 2024-04-24T16:01:16.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126700K .......... .......... .......... .......... .......... 75% 27.2M 1s 2024-04-24T16:01:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126750K .......... .......... .......... .......... .......... 75% 26.0M 1s 2024-04-24T16:01:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126800K .......... .......... .......... .......... .......... 75% 34.7M 1s 2024-04-24T16:01:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126850K .......... .......... .......... .......... .......... 75% 22.0M 1s 2024-04-24T16:01:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126900K .......... .......... .......... .......... .......... 75% 39.3M 1s 2024-04-24T16:01:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:126950K .......... .......... .......... .......... .......... 75% 27.6M 1s 2024-04-24T16:01:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127000K .......... .......... .......... .......... .......... 75% 18.0M 1s 2024-04-24T16:01:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127050K .......... .......... .......... .......... .......... 75% 35.5M 1s 2024-04-24T16:01:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127100K .......... .......... .......... .......... .......... 75% 40.6M 1s 2024-04-24T16:01:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127150K .......... .......... .......... .......... .......... 75% 35.1M 1s 2024-04-24T16:01:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127200K .......... .......... .......... .......... .......... 75% 34.7M 1s 2024-04-24T16:01:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127250K .......... .......... .......... .......... .......... 75% 18.4M 1s 2024-04-24T16:01:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127300K .......... .......... .......... .......... .......... 75% 54.0M 1s 2024-04-24T16:01:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127350K .......... .......... .......... .......... .......... 75% 21.6M 1s 2024-04-24T16:01:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127400K .......... .......... .......... .......... .......... 75% 44.5M 1s 2024-04-24T16:01:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127450K .......... .......... .......... .......... .......... 75% 24.3M 1s 2024-04-24T16:01:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127500K .......... .......... .......... .......... .......... 75% 40.3M 1s 2024-04-24T16:01:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127550K .......... .......... .......... .......... .......... 75% 15.9M 1s 2024-04-24T16:01:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127600K .......... .......... .......... .......... .......... 75% 56.5M 1s 2024-04-24T16:01:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127650K .......... .......... .......... .......... .......... 75% 23.3M 1s 2024-04-24T16:01:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127700K .......... .......... .......... .......... .......... 75% 43.0M 1s 2024-04-24T16:01:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127750K .......... .......... .......... .......... .......... 75% 26.0M 1s 2024-04-24T16:01:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127800K .......... .......... .......... .......... .......... 75% 45.0M 1s 2024-04-24T16:01:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127850K .......... .......... .......... .......... .......... 75% 31.1M 1s 2024-04-24T16:01:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127900K .......... .......... .......... .......... .......... 75% 35.5M 1s 2024-04-24T16:01:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:127950K .......... .......... .......... .......... .......... 75% 31.0M 1s 2024-04-24T16:01:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128000K .......... .......... .......... .......... .......... 75% 24.3M 1s 2024-04-24T16:01:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128050K .......... .......... .......... .......... .......... 75% 45.8M 1s 2024-04-24T16:01:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128100K .......... .......... .......... .......... .......... 76% 22.7M 1s 2024-04-24T16:01:16.499 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128150K .......... .......... .......... .......... .......... 76% 31.7M 1s 2024-04-24T16:01:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128200K .......... .......... .......... .......... .......... 76% 34.4M 1s 2024-04-24T16:01:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128250K .......... .......... .......... .......... .......... 76% 26.7M 1s 2024-04-24T16:01:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128300K .......... .......... .......... .......... .......... 76% 37.7M 1s 2024-04-24T16:01:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128350K .......... .......... .......... .......... .......... 76% 41.9M 1s 2024-04-24T16:01:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128400K .......... .......... .......... .......... .......... 76% 26.1M 1s 2024-04-24T16:01:16.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128450K .......... .......... .......... .......... .......... 76% 21.7M 1s 2024-04-24T16:01:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128500K .......... .......... .......... .......... .......... 76% 17.5M 1s 2024-04-24T16:01:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128550K .......... .......... .......... .......... .......... 76% 44.7M 1s 2024-04-24T16:01:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128600K .......... .......... .......... .......... .......... 76% 20.9M 1s 2024-04-24T16:01:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128650K .......... .......... .......... .......... .......... 76% 23.5M 1s 2024-04-24T16:01:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128700K .......... .......... .......... .......... .......... 76% 25.5M 1s 2024-04-24T16:01:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128750K .......... .......... .......... .......... .......... 76% 22.7M 1s 2024-04-24T16:01:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128800K .......... .......... .......... .......... .......... 76% 34.2M 1s 2024-04-24T16:01:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128850K .......... .......... .......... .......... .......... 76% 29.7M 1s 2024-04-24T16:01:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128900K .......... .......... .......... .......... .......... 76% 36.2M 1s 2024-04-24T16:01:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:128950K .......... .......... .......... .......... .......... 76% 26.0M 1s 2024-04-24T16:01:16.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129000K .......... .......... .......... .......... .......... 76% 34.6M 1s 2024-04-24T16:01:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129050K .......... .......... .......... .......... .......... 76% 28.7M 1s 2024-04-24T16:01:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129100K .......... .......... .......... .......... .......... 76% 25.1M 1s 2024-04-24T16:01:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129150K .......... .......... .......... .......... .......... 76% 21.5M 1s 2024-04-24T16:01:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129200K .......... .......... .......... .......... .......... 76% 24.9M 1s 2024-04-24T16:01:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129250K .......... .......... .......... .......... .......... 76% 23.4M 1s 2024-04-24T16:01:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129300K .......... .......... .......... .......... .......... 76% 26.6M 1s 2024-04-24T16:01:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129350K .......... .......... .......... .......... .......... 76% 41.8M 1s 2024-04-24T16:01:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129400K .......... .......... .......... .......... .......... 76% 36.4M 1s 2024-04-24T16:01:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129450K .......... .......... .......... .......... .......... 76% 31.1M 1s 2024-04-24T16:01:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129500K .......... .......... .......... .......... .......... 76% 33.5M 1s 2024-04-24T16:01:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129550K .......... .......... .......... .......... .......... 76% 29.6M 1s 2024-04-24T16:01:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129600K .......... .......... .......... .......... .......... 76% 28.3M 1s 2024-04-24T16:01:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129650K .......... .......... .......... .......... .......... 76% 21.1M 1s 2024-04-24T16:01:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129700K .......... .......... .......... .......... .......... 76% 34.4M 1s 2024-04-24T16:01:16.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129750K .......... .......... .......... .......... .......... 77% 24.2M 1s 2024-04-24T16:01:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129800K .......... .......... .......... .......... .......... 77% 22.5M 1s 2024-04-24T16:01:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129850K .......... .......... .......... .......... .......... 77% 51.2M 1s 2024-04-24T16:01:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129900K .......... .......... .......... .......... .......... 77% 21.7M 1s 2024-04-24T16:01:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:129950K .......... .......... .......... .......... .......... 77% 25.9M 1s 2024-04-24T16:01:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130000K .......... .......... .......... .......... .......... 77% 34.0M 1s 2024-04-24T16:01:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130050K .......... .......... .......... .......... .......... 77% 38.0M 1s 2024-04-24T16:01:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130100K .......... .......... .......... .......... .......... 77% 37.7M 1s 2024-04-24T16:01:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130150K .......... .......... .......... .......... .......... 77% 36.4M 1s 2024-04-24T16:01:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130200K .......... .......... .......... .......... .......... 77% 27.5M 1s 2024-04-24T16:01:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130250K .......... .......... .......... .......... .......... 77% 27.2M 1s 2024-04-24T16:01:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130300K .......... .......... .......... .......... .......... 77% 42.9M 1s 2024-04-24T16:01:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130350K .......... .......... .......... .......... .......... 77% 17.2M 1s 2024-04-24T16:01:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130400K .......... .......... .......... .......... .......... 77% 48.9M 1s 2024-04-24T16:01:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130450K .......... .......... .......... .......... .......... 77% 35.5M 1s 2024-04-24T16:01:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130500K .......... .......... .......... .......... .......... 77% 35.0M 1s 2024-04-24T16:01:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130550K .......... .......... .......... .......... .......... 77% 21.5M 1s 2024-04-24T16:01:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130600K .......... .......... .......... .......... .......... 77% 36.0M 1s 2024-04-24T16:01:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130650K .......... .......... .......... .......... .......... 77% 22.9M 1s 2024-04-24T16:01:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130700K .......... .......... .......... .......... .......... 77% 29.5M 1s 2024-04-24T16:01:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130750K .......... .......... .......... .......... .......... 77% 53.9M 1s 2024-04-24T16:01:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130800K .......... .......... .......... .......... .......... 77% 72.4M 1s 2024-04-24T16:01:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130850K .......... .......... .......... .......... .......... 77% 40.0M 1s 2024-04-24T16:01:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130900K .......... .......... .......... .......... .......... 77% 20.3M 1s 2024-04-24T16:01:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:130950K .......... .......... .......... .......... .......... 77% 33.0M 1s 2024-04-24T16:01:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131000K .......... .......... .......... .......... .......... 77% 24.5M 1s 2024-04-24T16:01:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131050K .......... .......... .......... .......... .......... 77% 27.7M 1s 2024-04-24T16:01:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131100K .......... .......... .......... .......... .......... 77% 25.4M 1s 2024-04-24T16:01:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131150K .......... .......... .......... .......... .......... 77% 25.3M 1s 2024-04-24T16:01:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131200K .......... .......... .......... .......... .......... 77% 23.9M 1s 2024-04-24T16:01:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131250K .......... .......... .......... .......... .......... 77% 36.5M 1s 2024-04-24T16:01:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131300K .......... .......... .......... .......... .......... 77% 28.7M 1s 2024-04-24T16:01:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131350K .......... .......... .......... .......... .......... 77% 35.6M 1s 2024-04-24T16:01:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131400K .......... .......... .......... .......... .......... 77% 24.0M 1s 2024-04-24T16:01:16.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131450K .......... .......... .......... .......... .......... 78% 35.8M 1s 2024-04-24T16:01:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131500K .......... .......... .......... .......... .......... 78% 30.6M 1s 2024-04-24T16:01:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131550K .......... .......... .......... .......... .......... 78% 18.1M 1s 2024-04-24T16:01:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131600K .......... .......... .......... .......... .......... 78% 23.7M 1s 2024-04-24T16:01:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131650K .......... .......... .......... .......... .......... 78% 32.9M 1s 2024-04-24T16:01:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131700K .......... .......... .......... .......... .......... 78% 33.1M 1s 2024-04-24T16:01:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131750K .......... .......... .......... .......... .......... 78% 69.3M 1s 2024-04-24T16:01:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131800K .......... .......... .......... .......... .......... 78% 20.6M 1s 2024-04-24T16:01:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131850K .......... .......... .......... .......... .......... 78% 66.3M 1s 2024-04-24T16:01:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131900K .......... .......... .......... .......... .......... 78% 22.1M 1s 2024-04-24T16:01:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:131950K .......... .......... .......... .......... .......... 78% 30.0M 1s 2024-04-24T16:01:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132000K .......... .......... .......... .......... .......... 78% 32.7M 1s 2024-04-24T16:01:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132050K .......... .......... .......... .......... .......... 78% 21.3M 1s 2024-04-24T16:01:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132100K .......... .......... .......... .......... .......... 78% 24.3M 1s 2024-04-24T16:01:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132150K .......... .......... .......... .......... .......... 78% 51.7M 1s 2024-04-24T16:01:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132200K .......... .......... .......... .......... .......... 78% 21.2M 1s 2024-04-24T16:01:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132250K .......... .......... .......... .......... .......... 78% 46.5M 1s 2024-04-24T16:01:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132300K .......... .......... .......... .......... .......... 78% 30.5M 1s 2024-04-24T16:01:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132350K .......... .......... .......... .......... .......... 78% 21.7M 1s 2024-04-24T16:01:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132400K .......... .......... .......... .......... .......... 78% 41.6M 1s 2024-04-24T16:01:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132450K .......... .......... .......... .......... .......... 78% 23.3M 1s 2024-04-24T16:01:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132500K .......... .......... .......... .......... .......... 78% 24.7M 1s 2024-04-24T16:01:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132550K .......... .......... .......... .......... .......... 78% 42.9M 1s 2024-04-24T16:01:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132600K .......... .......... .......... .......... .......... 78% 40.5M 1s 2024-04-24T16:01:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132650K .......... .......... .......... .......... .......... 78% 25.4M 1s 2024-04-24T16:01:16.653 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132700K .......... .......... .......... .......... .......... 78% 37.5M 1s 2024-04-24T16:01:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132750K .......... .......... .......... .......... .......... 78% 20.1M 1s 2024-04-24T16:01:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132800K .......... .......... .......... .......... .......... 78% 33.7M 1s 2024-04-24T16:01:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132850K .......... .......... .......... .......... .......... 78% 26.9M 1s 2024-04-24T16:01:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132900K .......... .......... .......... .......... .......... 78% 36.2M 1s 2024-04-24T16:01:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:132950K .......... .......... .......... .......... .......... 78% 24.7M 1s 2024-04-24T16:01:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133000K .......... .......... .......... .......... .......... 78% 44.7M 1s 2024-04-24T16:01:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133050K .......... .......... .......... .......... .......... 78% 31.9M 1s 2024-04-24T16:01:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133100K .......... .......... .......... .......... .......... 78% 49.8M 1s 2024-04-24T16:01:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133150K .......... .......... .......... .......... .......... 79% 29.0M 1s 2024-04-24T16:01:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133200K .......... .......... .......... .......... .......... 79% 47.8M 1s 2024-04-24T16:01:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133250K .......... .......... .......... .......... .......... 79% 34.6M 1s 2024-04-24T16:01:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133300K .......... .......... .......... .......... .......... 79% 64.6M 1s 2024-04-24T16:01:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133350K .......... .......... .......... .......... .......... 79% 21.0M 1s 2024-04-24T16:01:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133400K .......... .......... .......... .......... .......... 79% 37.2M 1s 2024-04-24T16:01:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133450K .......... .......... .......... .......... .......... 79% 22.3M 1s 2024-04-24T16:01:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133500K .......... .......... .......... .......... .......... 79% 20.1M 1s 2024-04-24T16:01:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133550K .......... .......... .......... .......... .......... 79% 30.7M 1s 2024-04-24T16:01:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133600K .......... .......... .......... .......... .......... 79% 17.2M 1s 2024-04-24T16:01:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133650K .......... .......... .......... .......... .......... 79% 36.2M 1s 2024-04-24T16:01:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133700K .......... .......... .......... .......... .......... 79% 22.1M 1s 2024-04-24T16:01:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133750K .......... .......... .......... .......... .......... 79% 26.0M 1s 2024-04-24T16:01:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133800K .......... .......... .......... .......... .......... 79% 22.9M 1s 2024-04-24T16:01:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133850K .......... .......... .......... .......... .......... 79% 29.8M 1s 2024-04-24T16:01:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133900K .......... .......... .......... .......... .......... 79% 45.3M 1s 2024-04-24T16:01:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:133950K .......... .......... .......... .......... .......... 79% 18.4M 1s 2024-04-24T16:01:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134000K .......... .......... .......... .......... .......... 79% 23.7M 1s 2024-04-24T16:01:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134050K .......... .......... .......... .......... .......... 79% 60.1M 1s 2024-04-24T16:01:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134100K .......... .......... .......... .......... .......... 79% 49.6M 1s 2024-04-24T16:01:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134150K .......... .......... .......... .......... .......... 79% 25.0M 1s 2024-04-24T16:01:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134200K .......... .......... .......... .......... .......... 79% 22.4M 1s 2024-04-24T16:01:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134250K .......... .......... .......... .......... .......... 79% 40.4M 1s 2024-04-24T16:01:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134300K .......... .......... .......... .......... .......... 79% 28.0M 1s 2024-04-24T16:01:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134350K .......... .......... .......... .......... .......... 79% 30.6M 1s 2024-04-24T16:01:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134400K .......... .......... .......... .......... .......... 79% 54.6M 1s 2024-04-24T16:01:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134450K .......... .......... .......... .......... .......... 79% 20.9M 1s 2024-04-24T16:01:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134500K .......... .......... .......... .......... .......... 79% 28.2M 1s 2024-04-24T16:01:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134550K .......... .......... .......... .......... .......... 79% 32.1M 1s 2024-04-24T16:01:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134600K .......... .......... .......... .......... .......... 79% 32.7M 1s 2024-04-24T16:01:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134650K .......... .......... .......... .......... .......... 79% 43.8M 1s 2024-04-24T16:01:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134700K .......... .......... .......... .......... .......... 79% 24.1M 1s 2024-04-24T16:01:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134750K .......... .......... .......... .......... .......... 79% 21.9M 1s 2024-04-24T16:01:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134800K .......... .......... .......... .......... .......... 79% 26.9M 1s 2024-04-24T16:01:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134850K .......... .......... .......... .......... .......... 80% 90.6M 1s 2024-04-24T16:01:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134900K .......... .......... .......... .......... .......... 80% 43.9M 1s 2024-04-24T16:01:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:134950K .......... .......... .......... .......... .......... 80% 57.6M 1s 2024-04-24T16:01:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135000K .......... .......... .......... .......... .......... 80% 46.5M 1s 2024-04-24T16:01:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135050K .......... .......... .......... .......... .......... 80% 68.2M 1s 2024-04-24T16:01:16.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135100K .......... .......... .......... .......... .......... 80% 51.4M 1s 2024-04-24T16:01:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135150K .......... .......... .......... .......... .......... 80% 26.3M 1s 2024-04-24T16:01:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135200K .......... .......... .......... .......... .......... 80% 35.0M 1s 2024-04-24T16:01:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135250K .......... .......... .......... .......... .......... 80% 28.2M 1s 2024-04-24T16:01:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135300K .......... .......... .......... .......... .......... 80% 28.1M 1s 2024-04-24T16:01:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135350K .......... .......... .......... .......... .......... 80% 25.6M 1s 2024-04-24T16:01:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135400K .......... .......... .......... .......... .......... 80% 63.6M 1s 2024-04-24T16:01:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135450K .......... .......... .......... .......... .......... 80% 22.8M 1s 2024-04-24T16:01:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135500K .......... .......... .......... .......... .......... 80% 36.2M 1s 2024-04-24T16:01:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135550K .......... .......... .......... .......... .......... 80% 22.3M 1s 2024-04-24T16:01:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135600K .......... .......... .......... .......... .......... 80% 20.8M 1s 2024-04-24T16:01:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135650K .......... .......... .......... .......... .......... 80% 25.1M 1s 2024-04-24T16:01:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135700K .......... .......... .......... .......... .......... 80% 29.3M 1s 2024-04-24T16:01:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135750K .......... .......... .......... .......... .......... 80% 19.5M 1s 2024-04-24T16:01:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135800K .......... .......... .......... .......... .......... 80% 25.1M 1s 2024-04-24T16:01:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135850K .......... .......... .......... .......... .......... 80% 20.3M 1s 2024-04-24T16:01:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135900K .......... .......... .......... .......... .......... 80% 30.2M 1s 2024-04-24T16:01:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:135950K .......... .......... .......... .......... .......... 80% 14.2M 1s 2024-04-24T16:01:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136000K .......... .......... .......... .......... .......... 80% 13.8M 1s 2024-04-24T16:01:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136050K .......... .......... .......... .......... .......... 80% 15.3M 1s 2024-04-24T16:01:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136100K .......... .......... .......... .......... .......... 80% 16.0M 1s 2024-04-24T16:01:16.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136150K .......... .......... .......... .......... .......... 80% 33.0M 1s 2024-04-24T16:01:16.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136200K .......... .......... .......... .......... .......... 80% 20.5M 1s 2024-04-24T16:01:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136250K .......... .......... .......... .......... .......... 80% 23.1M 1s 2024-04-24T16:01:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136300K .......... .......... .......... .......... .......... 80% 24.9M 1s 2024-04-24T16:01:16.791 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136350K .......... .......... .......... .......... .......... 80% 64.6M 1s 2024-04-24T16:01:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136400K .......... .......... .......... .......... .......... 80% 74.9M 1s 2024-04-24T16:01:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136450K .......... .......... .......... .......... .......... 80% 93.9M 1s 2024-04-24T16:01:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136500K .......... .......... .......... .......... .......... 81% 88.6M 1s 2024-04-24T16:01:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136550K .......... .......... .......... .......... .......... 81% 33.3M 1s 2024-04-24T16:01:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136600K .......... .......... .......... .......... .......... 81% 61.3M 1s 2024-04-24T16:01:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136650K .......... .......... .......... .......... .......... 81% 16.8M 1s 2024-04-24T16:01:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136700K .......... .......... .......... .......... .......... 81% 45.0M 1s 2024-04-24T16:01:16.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136750K .......... .......... .......... .......... .......... 81% 22.0M 1s 2024-04-24T16:01:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136800K .......... .......... .......... .......... .......... 81% 23.0M 1s 2024-04-24T16:01:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136850K .......... .......... .......... .......... .......... 81% 49.3M 1s 2024-04-24T16:01:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136900K .......... .......... .......... .......... .......... 81% 22.0M 1s 2024-04-24T16:01:16.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:136950K .......... .......... .......... .......... .......... 81% 46.1M 1s 2024-04-24T16:01:16.799 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137000K .......... .......... .......... .......... .......... 81% 25.3M 1s 2024-04-24T16:01:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137050K .......... .......... .......... .......... .......... 81% 31.3M 1s 2024-04-24T16:01:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137100K .......... .......... .......... .......... .......... 81% 22.9M 1s 2024-04-24T16:01:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137150K .......... .......... .......... .......... .......... 81% 19.0M 1s 2024-04-24T16:01:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137200K .......... .......... .......... .......... .......... 81% 23.6M 1s 2024-04-24T16:01:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137250K .......... .......... .......... .......... .......... 81% 33.3M 1s 2024-04-24T16:01:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137300K .......... .......... .......... .......... .......... 81% 22.3M 1s 2024-04-24T16:01:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137350K .......... .......... .......... .......... .......... 81% 34.6M 1s 2024-04-24T16:01:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137400K .......... .......... .......... .......... .......... 81% 20.7M 1s 2024-04-24T16:01:16.816 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137450K .......... .......... .......... .......... .......... 81% 39.2M 1s 2024-04-24T16:01:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137500K .......... .......... .......... .......... .......... 81% 24.7M 1s 2024-04-24T16:01:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137550K .......... .......... .......... .......... .......... 81% 30.5M 1s 2024-04-24T16:01:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137600K .......... .......... .......... .......... .......... 81% 23.0M 1s 2024-04-24T16:01:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137650K .......... .......... .......... .......... .......... 81% 27.5M 1s 2024-04-24T16:01:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137700K .......... .......... .......... .......... .......... 81% 43.0M 1s 2024-04-24T16:01:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137750K .......... .......... .......... .......... .......... 81% 50.1M 1s 2024-04-24T16:01:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137800K .......... .......... .......... .......... .......... 81% 29.4M 1s 2024-04-24T16:01:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137850K .......... .......... .......... .......... .......... 81% 39.9M 1s 2024-04-24T16:01:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137900K .......... .......... .......... .......... .......... 81% 24.3M 1s 2024-04-24T16:01:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:137950K .......... .......... .......... .......... .......... 81% 33.9M 1s 2024-04-24T16:01:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138000K .......... .......... .......... .......... .......... 81% 33.2M 1s 2024-04-24T16:01:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138050K .......... .......... .......... .......... .......... 81% 20.2M 1s 2024-04-24T16:01:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138100K .......... .......... .......... .......... .......... 81% 28.2M 1s 2024-04-24T16:01:16.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138150K .......... .......... .......... .......... .......... 81% 18.8M 1s 2024-04-24T16:01:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138200K .......... .......... .......... .......... .......... 82% 27.0M 1s 2024-04-24T16:01:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138250K .......... .......... .......... .......... .......... 82% 22.7M 1s 2024-04-24T16:01:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138300K .......... .......... .......... .......... .......... 82% 33.2M 1s 2024-04-24T16:01:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138350K .......... .......... .......... .......... .......... 82% 36.6M 1s 2024-04-24T16:01:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138400K .......... .......... .......... .......... .......... 82% 22.2M 1s 2024-04-24T16:01:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138450K .......... .......... .......... .......... .......... 82% 32.0M 1s 2024-04-24T16:01:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138500K .......... .......... .......... .......... .......... 82% 47.9M 1s 2024-04-24T16:01:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138550K .......... .......... .......... .......... .......... 82% 24.1M 1s 2024-04-24T16:01:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138600K .......... .......... .......... .......... .......... 82% 34.6M 1s 2024-04-24T16:01:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138650K .......... .......... .......... .......... .......... 82% 23.1M 1s 2024-04-24T16:01:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138700K .......... .......... .......... .......... .......... 82% 48.9M 1s 2024-04-24T16:01:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138750K .......... .......... .......... .......... .......... 82% 31.2M 1s 2024-04-24T16:01:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138800K .......... .......... .......... .......... .......... 82% 25.7M 1s 2024-04-24T16:01:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138850K .......... .......... .......... .......... .......... 82% 36.8M 1s 2024-04-24T16:01:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138900K .......... .......... .......... .......... .......... 82% 26.5M 1s 2024-04-24T16:01:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:138950K .......... .......... .......... .......... .......... 82% 26.8M 1s 2024-04-24T16:01:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139000K .......... .......... .......... .......... .......... 82% 51.6M 1s 2024-04-24T16:01:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139050K .......... .......... .......... .......... .......... 82% 30.6M 1s 2024-04-24T16:01:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139100K .......... .......... .......... .......... .......... 82% 32.8M 1s 2024-04-24T16:01:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139150K .......... .......... .......... .......... .......... 82% 27.1M 1s 2024-04-24T16:01:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139200K .......... .......... .......... .......... .......... 82% 25.9M 1s 2024-04-24T16:01:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139250K .......... .......... .......... .......... .......... 82% 33.1M 1s 2024-04-24T16:01:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139300K .......... .......... .......... .......... .......... 82% 29.2M 1s 2024-04-24T16:01:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139350K .......... .......... .......... .......... .......... 82% 20.6M 1s 2024-04-24T16:01:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139400K .......... .......... .......... .......... .......... 82% 36.3M 1s 2024-04-24T16:01:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139450K .......... .......... .......... .......... .......... 82% 24.3M 1s 2024-04-24T16:01:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139500K .......... .......... .......... .......... .......... 82% 38.4M 1s 2024-04-24T16:01:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139550K .......... .......... .......... .......... .......... 82% 22.1M 1s 2024-04-24T16:01:16.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139600K .......... .......... .......... .......... .......... 82% 22.0M 1s 2024-04-24T16:01:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139650K .......... .......... .......... .......... .......... 82% 43.2M 1s 2024-04-24T16:01:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139700K .......... .......... .......... .......... .......... 82% 36.8M 1s 2024-04-24T16:01:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139750K .......... .......... .......... .......... .......... 82% 29.1M 1s 2024-04-24T16:01:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139800K .......... .......... .......... .......... .......... 82% 31.1M 1s 2024-04-24T16:01:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139850K .......... .......... .......... .......... .......... 82% 32.6M 1s 2024-04-24T16:01:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139900K .......... .......... .......... .......... .......... 83% 25.3M 1s 2024-04-24T16:01:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:139950K .......... .......... .......... .......... .......... 83% 25.1M 1s 2024-04-24T16:01:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140000K .......... .......... .......... .......... .......... 83% 30.1M 1s 2024-04-24T16:01:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140050K .......... .......... .......... .......... .......... 83% 42.2M 1s 2024-04-24T16:01:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140100K .......... .......... .......... .......... .......... 83% 26.6M 1s 2024-04-24T16:01:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140150K .......... .......... .......... .......... .......... 83% 37.5M 1s 2024-04-24T16:01:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140200K .......... .......... .......... .......... .......... 83% 28.4M 1s 2024-04-24T16:01:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140250K .......... .......... .......... .......... .......... 83% 57.3M 1s 2024-04-24T16:01:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140300K .......... .......... .......... .......... .......... 83% 31.3M 1s 2024-04-24T16:01:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140350K .......... .......... .......... .......... .......... 83% 30.7M 1s 2024-04-24T16:01:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140400K .......... .......... .......... .......... .......... 83% 41.0M 1s 2024-04-24T16:01:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140450K .......... .......... .......... .......... .......... 83% 23.7M 1s 2024-04-24T16:01:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140500K .......... .......... .......... .......... .......... 83% 46.4M 1s 2024-04-24T16:01:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140550K .......... .......... .......... .......... .......... 83% 17.2M 1s 2024-04-24T16:01:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140600K .......... .......... .......... .......... .......... 83% 24.0M 1s 2024-04-24T16:01:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140650K .......... .......... .......... .......... .......... 83% 19.3M 1s 2024-04-24T16:01:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140700K .......... .......... .......... .......... .......... 83% 14.3M 1s 2024-04-24T16:01:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140750K .......... .......... .......... .......... .......... 83% 22.0M 1s 2024-04-24T16:01:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140800K .......... .......... .......... .......... .......... 83% 45.6M 1s 2024-04-24T16:01:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140850K .......... .......... .......... .......... .......... 83% 33.5M 1s 2024-04-24T16:01:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140900K .......... .......... .......... .......... .......... 83% 21.2M 1s 2024-04-24T16:01:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:140950K .......... .......... .......... .......... .......... 83% 43.4M 1s 2024-04-24T16:01:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141000K .......... .......... .......... .......... .......... 83% 40.0M 1s 2024-04-24T16:01:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141050K .......... .......... .......... .......... .......... 83% 26.5M 1s 2024-04-24T16:01:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141100K .......... .......... .......... .......... .......... 83% 49.6M 1s 2024-04-24T16:01:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141150K .......... .......... .......... .......... .......... 83% 24.5M 1s 2024-04-24T16:01:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141200K .......... .......... .......... .......... .......... 83% 23.3M 1s 2024-04-24T16:01:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141250K .......... .......... .......... .......... .......... 83% 32.1M 1s 2024-04-24T16:01:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141300K .......... .......... .......... .......... .......... 83% 40.3M 1s 2024-04-24T16:01:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141350K .......... .......... .......... .......... .......... 83% 52.0M 1s 2024-04-24T16:01:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141400K .......... .......... .......... .......... .......... 83% 117M 1s 2024-04-24T16:01:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141450K .......... .......... .......... .......... .......... 83% 105M 1s 2024-04-24T16:01:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141500K .......... .......... .......... .......... .......... 83% 73.9M 1s 2024-04-24T16:01:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141550K .......... .......... .......... .......... .......... 84% 42.8M 1s 2024-04-24T16:01:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141600K .......... .......... .......... .......... .......... 84% 55.7M 1s 2024-04-24T16:01:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141650K .......... .......... .......... .......... .......... 84% 92.1M 1s 2024-04-24T16:01:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141700K .......... .......... .......... .......... .......... 84% 31.9M 1s 2024-04-24T16:01:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141750K .......... .......... .......... .......... .......... 84% 66.5M 1s 2024-04-24T16:01:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141800K .......... .......... .......... .......... .......... 84% 25.1M 1s 2024-04-24T16:01:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141850K .......... .......... .......... .......... .......... 84% 30.7M 1s 2024-04-24T16:01:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141900K .......... .......... .......... .......... .......... 84% 27.6M 1s 2024-04-24T16:01:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:141950K .......... .......... .......... .......... .......... 84% 18.9M 1s 2024-04-24T16:01:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142000K .......... .......... .......... .......... .......... 84% 31.8M 1s 2024-04-24T16:01:16.963 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142050K .......... .......... .......... .......... .......... 84% 49.5M 1s 2024-04-24T16:01:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142100K .......... .......... .......... .......... .......... 84% 26.1M 1s 2024-04-24T16:01:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142150K .......... .......... .......... .......... .......... 84% 32.4M 1s 2024-04-24T16:01:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142200K .......... .......... .......... .......... .......... 84% 30.0M 1s 2024-04-24T16:01:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142250K .......... .......... .......... .......... .......... 84% 20.4M 1s 2024-04-24T16:01:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142300K .......... .......... .......... .......... .......... 84% 61.5M 1s 2024-04-24T16:01:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142350K .......... .......... .......... .......... .......... 84% 22.6M 1s 2024-04-24T16:01:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142400K .......... .......... .......... .......... .......... 84% 28.7M 1s 2024-04-24T16:01:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142450K .......... .......... .......... .......... .......... 84% 26.1M 1s 2024-04-24T16:01:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142500K .......... .......... .......... .......... .......... 84% 25.0M 1s 2024-04-24T16:01:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142550K .......... .......... .......... .......... .......... 84% 50.4M 1s 2024-04-24T16:01:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142600K .......... .......... .......... .......... .......... 84% 24.0M 1s 2024-04-24T16:01:16.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142650K .......... .......... .......... .......... .......... 84% 35.8M 1s 2024-04-24T16:01:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142700K .......... .......... .......... .......... .......... 84% 24.7M 1s 2024-04-24T16:01:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142750K .......... .......... .......... .......... .......... 84% 23.6M 1s 2024-04-24T16:01:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142800K .......... .......... .......... .......... .......... 84% 24.3M 1s 2024-04-24T16:01:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142850K .......... .......... .......... .......... .......... 84% 48.4M 1s 2024-04-24T16:01:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142900K .......... .......... .......... .......... .......... 84% 35.7M 1s 2024-04-24T16:01:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:142950K .......... .......... .......... .......... .......... 84% 24.1M 1s 2024-04-24T16:01:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143000K .......... .......... .......... .......... .......... 84% 27.5M 1s 2024-04-24T16:01:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143050K .......... .......... .......... .......... .......... 84% 47.9M 1s 2024-04-24T16:01:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143100K .......... .......... .......... .......... .......... 84% 24.7M 1s 2024-04-24T16:01:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143150K .......... .......... .......... .......... .......... 84% 23.5M 1s 2024-04-24T16:01:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143200K .......... .......... .......... .......... .......... 84% 23.3M 1s 2024-04-24T16:01:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143250K .......... .......... .......... .......... .......... 85% 34.3M 1s 2024-04-24T16:01:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143300K .......... .......... .......... .......... .......... 85% 54.1M 1s 2024-04-24T16:01:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143350K .......... .......... .......... .......... .......... 85% 22.4M 1s 2024-04-24T16:01:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143400K .......... .......... .......... .......... .......... 85% 37.6M 1s 2024-04-24T16:01:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143450K .......... .......... .......... .......... .......... 85% 25.2M 1s 2024-04-24T16:01:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143500K .......... .......... .......... .......... .......... 85% 23.5M 1s 2024-04-24T16:01:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143550K .......... .......... .......... .......... .......... 85% 23.6M 1s 2024-04-24T16:01:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143600K .......... .......... .......... .......... .......... 85% 58.5M 1s 2024-04-24T16:01:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143650K .......... .......... .......... .......... .......... 85% 24.5M 1s 2024-04-24T16:01:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143700K .......... .......... .......... .......... .......... 85% 29.9M 1s 2024-04-24T16:01:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143750K .......... .......... .......... .......... .......... 85% 25.5M 1s 2024-04-24T16:01:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143800K .......... .......... .......... .......... .......... 85% 43.6M 1s 2024-04-24T16:01:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143850K .......... .......... .......... .......... .......... 85% 19.2M 1s 2024-04-24T16:01:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143900K .......... .......... .......... .......... .......... 85% 29.2M 1s 2024-04-24T16:01:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:143950K .......... .......... .......... .......... .......... 85% 18.9M 1s 2024-04-24T16:01:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144000K .......... .......... .......... .......... .......... 85% 82.3M 1s 2024-04-24T16:01:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144050K .......... .......... .......... .......... .......... 85% 25.0M 1s 2024-04-24T16:01:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144100K .......... .......... .......... .......... .......... 85% 49.0M 1s 2024-04-24T16:01:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144150K .......... .......... .......... .......... .......... 85% 25.0M 1s 2024-04-24T16:01:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144200K .......... .......... .......... .......... .......... 85% 26.7M 1s 2024-04-24T16:01:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144250K .......... .......... .......... .......... .......... 85% 30.0M 1s 2024-04-24T16:01:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144300K .......... .......... .......... .......... .......... 85% 49.9M 1s 2024-04-24T16:01:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144350K .......... .......... .......... .......... .......... 85% 23.0M 1s 2024-04-24T16:01:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144400K .......... .......... .......... .......... .......... 85% 24.7M 1s 2024-04-24T16:01:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144450K .......... .......... .......... .......... .......... 85% 29.5M 1s 2024-04-24T16:01:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144500K .......... .......... .......... .......... .......... 85% 24.5M 1s 2024-04-24T16:01:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144550K .......... .......... .......... .......... .......... 85% 28.4M 1s 2024-04-24T16:01:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144600K .......... .......... .......... .......... .......... 85% 42.1M 1s 2024-04-24T16:01:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144650K .......... .......... .......... .......... .......... 85% 33.0M 1s 2024-04-24T16:01:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144700K .......... .......... .......... .......... .......... 85% 34.0M 1s 2024-04-24T16:01:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144750K .......... .......... .......... .......... .......... 85% 19.0M 1s 2024-04-24T16:01:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144800K .......... .......... .......... .......... .......... 85% 23.7M 1s 2024-04-24T16:01:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144850K .......... .......... .......... .......... .......... 85% 59.0M 1s 2024-04-24T16:01:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144900K .......... .......... .......... .......... .......... 85% 23.8M 1s 2024-04-24T16:01:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:144950K .......... .......... .......... .......... .......... 86% 32.5M 1s 2024-04-24T16:01:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145000K .......... .......... .......... .......... .......... 86% 24.7M 1s 2024-04-24T16:01:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145050K .......... .......... .......... .......... .......... 86% 76.7M 1s 2024-04-24T16:01:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145100K .......... .......... .......... .......... .......... 86% 22.9M 1s 2024-04-24T16:01:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145150K .......... .......... .......... .......... .......... 86% 23.0M 1s 2024-04-24T16:01:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145200K .......... .......... .......... .......... .......... 86% 33.6M 1s 2024-04-24T16:01:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145250K .......... .......... .......... .......... .......... 86% 25.3M 1s 2024-04-24T16:01:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145300K .......... .......... .......... .......... .......... 86% 24.9M 1s 2024-04-24T16:01:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145350K .......... .......... .......... .......... .......... 86% 49.0M 1s 2024-04-24T16:01:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145400K .......... .......... .......... .......... .......... 86% 24.1M 1s 2024-04-24T16:01:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145450K .......... .......... .......... .......... .......... 86% 35.1M 1s 2024-04-24T16:01:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145500K .......... .......... .......... .......... .......... 86% 23.7M 1s 2024-04-24T16:01:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145550K .......... .......... .......... .......... .......... 86% 31.7M 1s 2024-04-24T16:01:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145600K .......... .......... .......... .......... .......... 86% 32.5M 1s 2024-04-24T16:01:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145650K .......... .......... .......... .......... .......... 86% 26.0M 1s 2024-04-24T16:01:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145700K .......... .......... .......... .......... .......... 86% 23.8M 1s 2024-04-24T16:01:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145750K .......... .......... .......... .......... .......... 86% 28.5M 1s 2024-04-24T16:01:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145800K .......... .......... .......... .......... .......... 86% 34.9M 1s 2024-04-24T16:01:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145850K .......... .......... .......... .......... .......... 86% 42.3M 1s 2024-04-24T16:01:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145900K .......... .......... .......... .......... .......... 86% 32.2M 1s 2024-04-24T16:01:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:145950K .......... .......... .......... .......... .......... 86% 15.8M 1s 2024-04-24T16:01:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146000K .......... .......... .......... .......... .......... 86% 27.4M 1s 2024-04-24T16:01:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146050K .......... .......... .......... .......... .......... 86% 46.4M 1s 2024-04-24T16:01:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146100K .......... .......... .......... .......... .......... 86% 44.6M 1s 2024-04-24T16:01:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146150K .......... .......... .......... .......... .......... 86% 24.4M 1s 2024-04-24T16:01:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146200K .......... .......... .......... .......... .......... 86% 30.6M 1s 2024-04-24T16:01:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146250K .......... .......... .......... .......... .......... 86% 21.3M 1s 2024-04-24T16:01:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146300K .......... .......... .......... .......... .......... 86% 77.2M 1s 2024-04-24T16:01:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146350K .......... .......... .......... .......... .......... 86% 25.7M 1s 2024-04-24T16:01:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146400K .......... .......... .......... .......... .......... 86% 25.3M 1s 2024-04-24T16:01:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146450K .......... .......... .......... .......... .......... 86% 29.3M 1s 2024-04-24T16:01:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146500K .......... .......... .......... .......... .......... 86% 28.3M 1s 2024-04-24T16:01:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146550K .......... .......... .......... .......... .......... 86% 22.9M 1s 2024-04-24T16:01:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146600K .......... .......... .......... .......... .......... 86% 57.2M 1s 2024-04-24T16:01:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146650K .......... .......... .......... .......... .......... 87% 25.1M 1s 2024-04-24T16:01:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146700K .......... .......... .......... .......... .......... 87% 24.4M 1s 2024-04-24T16:01:17.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146750K .......... .......... .......... .......... .......... 87% 29.8M 1s 2024-04-24T16:01:17.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146800K .......... .......... .......... .......... .......... 87% 23.8M 1s 2024-04-24T16:01:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146850K .......... .......... .......... .......... .......... 87% 58.7M 1s 2024-04-24T16:01:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146900K .......... .......... .......... .......... .......... 87% 24.4M 1s 2024-04-24T16:01:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:146950K .......... .......... .......... .......... .......... 87% 33.0M 1s 2024-04-24T16:01:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147000K .......... .......... .......... .......... .......... 87% 24.4M 1s 2024-04-24T16:01:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147050K .......... .......... .......... .......... .......... 87% 27.9M 1s 2024-04-24T16:01:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147100K .......... .......... .......... .......... .......... 87% 47.2M 1s 2024-04-24T16:01:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147150K .......... .......... .......... .......... .......... 87% 23.9M 1s 2024-04-24T16:01:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147200K .......... .......... .......... .......... .......... 87% 23.5M 1s 2024-04-24T16:01:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147250K .......... .......... .......... .......... .......... 87% 31.4M 1s 2024-04-24T16:01:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147300K .......... .......... .......... .......... .......... 87% 35.9M 1s 2024-04-24T16:01:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147350K .......... .......... .......... .......... .......... 87% 28.7M 1s 2024-04-24T16:01:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147400K .......... .......... .......... .......... .......... 87% 35.4M 1s 2024-04-24T16:01:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147450K .......... .......... .......... .......... .......... 87% 22.6M 1s 2024-04-24T16:01:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147500K .......... .......... .......... .......... .......... 87% 49.3M 1s 2024-04-24T16:01:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147550K .......... .......... .......... .......... .......... 87% 16.1M 1s 2024-04-24T16:01:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147600K .......... .......... .......... .......... .......... 87% 50.0M 1s 2024-04-24T16:01:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147650K .......... .......... .......... .......... .......... 87% 29.3M 1s 2024-04-24T16:01:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147700K .......... .......... .......... .......... .......... 87% 28.7M 1s 2024-04-24T16:01:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147750K .......... .......... .......... .......... .......... 87% 23.5M 1s 2024-04-24T16:01:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147800K .......... .......... .......... .......... .......... 87% 13.3M 1s 2024-04-24T16:01:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147850K .......... .......... .......... .......... .......... 87% 203M 1s 2024-04-24T16:01:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147900K .......... .......... .......... .......... .......... 87% 58.2M 1s 2024-04-24T16:01:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:147950K .......... .......... .......... .......... .......... 87% 24.3M 1s 2024-04-24T16:01:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148000K .......... .......... .......... .......... .......... 87% 27.6M 1s 2024-04-24T16:01:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148050K .......... .......... .......... .......... .......... 87% 27.1M 1s 2024-04-24T16:01:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148100K .......... .......... .......... .......... .......... 87% 23.6M 1s 2024-04-24T16:01:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148150K .......... .......... .......... .......... .......... 87% 45.2M 1s 2024-04-24T16:01:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148200K .......... .......... .......... .......... .......... 87% 22.3M 1s 2024-04-24T16:01:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148250K .......... .......... .......... .......... .......... 87% 33.5M 1s 2024-04-24T16:01:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148300K .......... .......... .......... .......... .......... 88% 43.4M 1s 2024-04-24T16:01:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148350K .......... .......... .......... .......... .......... 88% 23.4M 1s 2024-04-24T16:01:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148400K .......... .......... .......... .......... .......... 88% 33.9M 1s 2024-04-24T16:01:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148450K .......... .......... .......... .......... .......... 88% 23.8M 1s 2024-04-24T16:01:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148500K .......... .......... .......... .......... .......... 88% 26.4M 1s 2024-04-24T16:01:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148550K .......... .......... .......... .......... .......... 88% 30.5M 1s 2024-04-24T16:01:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148600K .......... .......... .......... .......... .......... 88% 38.0M 1s 2024-04-24T16:01:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148650K .......... .......... .......... .......... .......... 88% 31.1M 1s 2024-04-24T16:01:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148700K .......... .......... .......... .......... .......... 88% 22.5M 1s 2024-04-24T16:01:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148750K .......... .......... .......... .......... .......... 88% 23.4M 1s 2024-04-24T16:01:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148800K .......... .......... .......... .......... .......... 88% 40.8M 1s 2024-04-24T16:01:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148850K .......... .......... .......... .......... .......... 88% 28.2M 1s 2024-04-24T16:01:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148900K .......... .......... .......... .......... .......... 88% 35.4M 1s 2024-04-24T16:01:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:148950K .......... .......... .......... .......... .......... 88% 24.3M 1s 2024-04-24T16:01:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149000K .......... .......... .......... .......... .......... 88% 35.4M 1s 2024-04-24T16:01:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149050K .......... .......... .......... .......... .......... 88% 31.9M 1s 2024-04-24T16:01:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149100K .......... .......... .......... .......... .......... 88% 23.2M 1s 2024-04-24T16:01:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149150K .......... .......... .......... .......... .......... 88% 19.6M 1s 2024-04-24T16:01:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149200K .......... .......... .......... .......... .......... 88% 55.4M 1s 2024-04-24T16:01:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149250K .......... .......... .......... .......... .......... 88% 24.0M 1s 2024-04-24T16:01:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149300K .......... .......... .......... .......... .......... 88% 45.0M 1s 2024-04-24T16:01:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149350K .......... .......... .......... .......... .......... 88% 24.0M 1s 2024-04-24T16:01:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149400K .......... .......... .......... .......... .......... 88% 36.8M 1s 2024-04-24T16:01:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149450K .......... .......... .......... .......... .......... 88% 25.4M 1s 2024-04-24T16:01:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149500K .......... .......... .......... .......... .......... 88% 39.6M 1s 2024-04-24T16:01:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149550K .......... .......... .......... .......... .......... 88% 18.1M 1s 2024-04-24T16:01:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149600K .......... .......... .......... .......... .......... 88% 34.7M 1s 2024-04-24T16:01:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149650K .......... .......... .......... .......... .......... 88% 30.1M 1s 2024-04-24T16:01:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149700K .......... .......... .......... .......... .......... 88% 38.7M 1s 2024-04-24T16:01:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149750K .......... .......... .......... .......... .......... 88% 21.4M 1s 2024-04-24T16:01:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149800K .......... .......... .......... .......... .......... 88% 58.4M 1s 2024-04-24T16:01:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149850K .......... .......... .......... .......... .......... 88% 19.5M 1s 2024-04-24T16:01:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149900K .......... .......... .......... .......... .......... 88% 52.3M 1s 2024-04-24T16:01:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:149950K .......... .......... .......... .......... .......... 88% 24.5M 1s 2024-04-24T16:01:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150000K .......... .......... .......... .......... .......... 89% 31.3M 1s 2024-04-24T16:01:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150050K .......... .......... .......... .......... .......... 89% 25.8M 1s 2024-04-24T16:01:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150100K .......... .......... .......... .......... .......... 89% 25.2M 1s 2024-04-24T16:01:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150150K .......... .......... .......... .......... .......... 89% 47.3M 1s 2024-04-24T16:01:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150200K .......... .......... .......... .......... .......... 89% 23.6M 1s 2024-04-24T16:01:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150250K .......... .......... .......... .......... .......... 89% 37.4M 1s 2024-04-24T16:01:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150300K .......... .......... .......... .......... .......... 89% 24.8M 1s 2024-04-24T16:01:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150350K .......... .......... .......... .......... .......... 89% 22.1M 1s 2024-04-24T16:01:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150400K .......... .......... .......... .......... .......... 89% 27.2M 1s 2024-04-24T16:01:17.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150450K .......... .......... .......... .......... .......... 89% 47.9M 1s 2024-04-24T16:01:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150500K .......... .......... .......... .......... .......... 89% 23.3M 1s 2024-04-24T16:01:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150550K .......... .......... .......... .......... .......... 89% 24.7M 1s 2024-04-24T16:01:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150600K .......... .......... .......... .......... .......... 89% 46.0M 1s 2024-04-24T16:01:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150650K .......... .......... .......... .......... .......... 89% 37.3M 1s 2024-04-24T16:01:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150700K .......... .......... .......... .......... .......... 89% 24.0M 1s 2024-04-24T16:01:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150750K .......... .......... .......... .......... .......... 89% 25.2M 1s 2024-04-24T16:01:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150800K .......... .......... .......... .......... .......... 89% 22.9M 1s 2024-04-24T16:01:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150850K .......... .......... .......... .......... .......... 89% 55.0M 1s 2024-04-24T16:01:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150900K .......... .......... .......... .......... .......... 89% 20.3M 1s 2024-04-24T16:01:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:150950K .......... .......... .......... .......... .......... 89% 46.1M 1s 2024-04-24T16:01:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151000K .......... .......... .......... .......... .......... 89% 31.9M 1s 2024-04-24T16:01:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151050K .......... .......... .......... .......... .......... 89% 30.8M 1s 2024-04-24T16:01:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151100K .......... .......... .......... .......... .......... 89% 23.0M 1s 2024-04-24T16:01:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151150K .......... .......... .......... .......... .......... 89% 25.0M 1s 2024-04-24T16:01:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151200K .......... .......... .......... .......... .......... 89% 42.2M 1s 2024-04-24T16:01:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151250K .......... .......... .......... .......... .......... 89% 23.7M 1s 2024-04-24T16:01:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151300K .......... .......... .......... .......... .......... 89% 35.8M 1s 2024-04-24T16:01:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151350K .......... .......... .......... .......... .......... 89% 25.8M 1s 2024-04-24T16:01:17.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151400K .......... .......... .......... .......... .......... 89% 25.1M 1s 2024-04-24T16:01:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151450K .......... .......... .......... .......... .......... 89% 50.1M 1s 2024-04-24T16:01:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151500K .......... .......... .......... .......... .......... 89% 23.1M 1s 2024-04-24T16:01:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151550K .......... .......... .......... .......... .......... 89% 26.2M 1s 2024-04-24T16:01:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151600K .......... .......... .......... .......... .......... 89% 32.3M 1s 2024-04-24T16:01:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151650K .......... .......... .......... .......... .......... 89% 20.1M 1s 2024-04-24T16:01:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151700K .......... .......... .......... .......... .......... 90% 84.6M 1s 2024-04-24T16:01:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151750K .......... .......... .......... .......... .......... 90% 24.2M 1s 2024-04-24T16:01:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151800K .......... .......... .......... .......... .......... 90% 37.4M 1s 2024-04-24T16:01:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151850K .......... .......... .......... .......... .......... 90% 25.1M 1s 2024-04-24T16:01:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151900K .......... .......... .......... .......... .......... 90% 24.0M 1s 2024-04-24T16:01:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:151950K .......... .......... .......... .......... .......... 90% 22.3M 1s 2024-04-24T16:01:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152000K .......... .......... .......... .......... .......... 90% 46.8M 1s 2024-04-24T16:01:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152050K .......... .......... .......... .......... .......... 90% 28.4M 1s 2024-04-24T16:01:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152100K .......... .......... .......... .......... .......... 90% 31.6M 1s 2024-04-24T16:01:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152150K .......... .......... .......... .......... .......... 90% 23.9M 1s 2024-04-24T16:01:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152200K .......... .......... .......... .......... .......... 90% 51.3M 1s 2024-04-24T16:01:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152250K .......... .......... .......... .......... .......... 90% 23.0M 1s 2024-04-24T16:01:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152300K .......... .......... .......... .......... .......... 90% 36.5M 1s 2024-04-24T16:01:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152350K .......... .......... .......... .......... .......... 90% 23.7M 1s 2024-04-24T16:01:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152400K .......... .......... .......... .......... .......... 90% 24.6M 1s 2024-04-24T16:01:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152450K .......... .......... .......... .......... .......... 90% 24.7M 1s 2024-04-24T16:01:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152500K .......... .......... .......... .......... .......... 90% 45.7M 1s 2024-04-24T16:01:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152550K .......... .......... .......... .......... .......... 90% 24.6M 1s 2024-04-24T16:01:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152600K .......... .......... .......... .......... .......... 90% 38.3M 1s 2024-04-24T16:01:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152650K .......... .......... .......... .......... .......... 90% 25.0M 1s 2024-04-24T16:01:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152700K .......... .......... .......... .......... .......... 90% 46.2M 1s 2024-04-24T16:01:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152750K .......... .......... .......... .......... .......... 90% 23.1M 1s 2024-04-24T16:01:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152800K .......... .......... .......... .......... .......... 90% 24.9M 1s 2024-04-24T16:01:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152850K .......... .......... .......... .......... .......... 90% 34.9M 1s 2024-04-24T16:01:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152900K .......... .......... .......... .......... .......... 90% 24.3M 1s 2024-04-24T16:01:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:152950K .......... .......... .......... .......... .......... 90% 47.1M 1s 2024-04-24T16:01:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153000K .......... .......... .......... .......... .......... 90% 21.7M 1s 2024-04-24T16:01:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153050K .......... .......... .......... .......... .......... 90% 32.0M 1s 2024-04-24T16:01:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153100K .......... .......... .......... .......... .......... 90% 32.3M 1s 2024-04-24T16:01:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153150K .......... .......... .......... .......... .......... 90% 23.7M 1s 2024-04-24T16:01:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153200K .......... .......... .......... .......... .......... 90% 39.5M 1s 2024-04-24T16:01:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153250K .......... .......... .......... .......... .......... 90% 27.4M 1s 2024-04-24T16:01:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153300K .......... .......... .......... .......... .......... 90% 28.1M 1s 2024-04-24T16:01:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153350K .......... .......... .......... .......... .......... 91% 33.7M 1s 2024-04-24T16:01:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153400K .......... .......... .......... .......... .......... 91% 23.2M 1s 2024-04-24T16:01:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153450K .......... .......... .......... .......... .......... 91% 48.7M 1s 2024-04-24T16:01:17.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153500K .......... .......... .......... .......... .......... 91% 32.5M 1s 2024-04-24T16:01:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153550K .......... .......... .......... .......... .......... 91% 20.3M 1s 2024-04-24T16:01:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153600K .......... .......... .......... .......... .......... 91% 30.5M 1s 2024-04-24T16:01:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153650K .......... .......... .......... .......... .......... 91% 26.3M 1s 2024-04-24T16:01:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153700K .......... .......... .......... .......... .......... 91% 23.4M 1s 2024-04-24T16:01:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153750K .......... .......... .......... .......... .......... 91% 69.7M 1s 2024-04-24T16:01:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153800K .......... .......... .......... .......... .......... 91% 24.1M 1s 2024-04-24T16:01:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153850K .......... .......... .......... .......... .......... 91% 31.0M 1s 2024-04-24T16:01:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153900K .......... .......... .......... .......... .......... 91% 22.7M 1s 2024-04-24T16:01:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:153950K .......... .......... .......... .......... .......... 91% 25.4M 1s 2024-04-24T16:01:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154000K .......... .......... .......... .......... .......... 91% 45.1M 1s 2024-04-24T16:01:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154050K .......... .......... .......... .......... .......... 91% 23.1M 1s 2024-04-24T16:01:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154100K .......... .......... .......... .......... .......... 91% 31.7M 1s 2024-04-24T16:01:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154150K .......... .......... .......... .......... .......... 91% 31.0M 1s 2024-04-24T16:01:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154200K .......... .......... .......... .......... .......... 91% 23.7M 1s 2024-04-24T16:01:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154250K .......... .......... .......... .......... .......... 91% 44.1M 0s 2024-04-24T16:01:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154300K .......... .......... .......... .......... .......... 91% 32.2M 0s 2024-04-24T16:01:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154350K .......... .......... .......... .......... .......... 91% 26.1M 0s 2024-04-24T16:01:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154400K .......... .......... .......... .......... .......... 91% 25.0M 0s 2024-04-24T16:01:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154450K .......... .......... .......... .......... .......... 91% 24.7M 0s 2024-04-24T16:01:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154500K .......... .......... .......... .......... .......... 91% 49.8M 0s 2024-04-24T16:01:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154550K .......... .......... .......... .......... .......... 91% 32.3M 0s 2024-04-24T16:01:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154600K .......... .......... .......... .......... .......... 91% 25.0M 0s 2024-04-24T16:01:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154650K .......... .......... .......... .......... .......... 91% 24.2M 0s 2024-04-24T16:01:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154700K .......... .......... .......... .......... .......... 91% 23.7M 0s 2024-04-24T16:01:17.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154750K .......... .......... .......... .......... .......... 91% 24.2M 0s 2024-04-24T16:01:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154800K .......... .......... .......... .......... .......... 91% 46.5M 0s 2024-04-24T16:01:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154850K .......... .......... .......... .......... .......... 91% 30.2M 0s 2024-04-24T16:01:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154900K .......... .......... .......... .......... .......... 91% 29.5M 0s 2024-04-24T16:01:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:154950K .......... .......... .......... .......... .......... 91% 22.9M 0s 2024-04-24T16:01:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155000K .......... .......... .......... .......... .......... 91% 56.1M 0s 2024-04-24T16:01:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155050K .......... .......... .......... .......... .......... 92% 25.6M 0s 2024-04-24T16:01:17.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155100K .......... .......... .......... .......... .......... 92% 34.8M 0s 2024-04-24T16:01:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155150K .......... .......... .......... .......... .......... 92% 23.3M 0s 2024-04-24T16:01:17.408 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155200K .......... .......... .......... .......... .......... 92% 24.2M 0s 2024-04-24T16:01:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155250K .......... .......... .......... .......... .......... 92% 44.5M 0s 2024-04-24T16:01:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155300K .......... .......... .......... .......... .......... 92% 29.6M 0s 2024-04-24T16:01:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155350K .......... .......... .......... .......... .......... 92% 28.8M 0s 2024-04-24T16:01:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155400K .......... .......... .......... .......... .......... 92% 24.1M 0s 2024-04-24T16:01:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155450K .......... .......... .......... .......... .......... 92% 25.6M 0s 2024-04-24T16:01:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155500K .......... .......... .......... .......... .......... 92% 48.9M 0s 2024-04-24T16:01:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155550K .......... .......... .......... .......... .......... 92% 24.2M 0s 2024-04-24T16:01:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155600K .......... .......... .......... .......... .......... 92% 23.3M 0s 2024-04-24T16:01:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155650K .......... .......... .......... .......... .......... 92% 37.5M 0s 2024-04-24T16:01:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155700K .......... .......... .......... .......... .......... 92% 25.3M 0s 2024-04-24T16:01:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155750K .......... .......... .......... .......... .......... 92% 50.4M 0s 2024-04-24T16:01:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155800K .......... .......... .......... .......... .......... 92% 29.4M 0s 2024-04-24T16:01:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155850K .......... .......... .......... .......... .......... 92% 28.9M 0s 2024-04-24T16:01:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155900K .......... .......... .......... .......... .......... 92% 28.1M 0s 2024-04-24T16:01:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:155950K .......... .......... .......... .......... .......... 92% 19.5M 0s 2024-04-24T16:01:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156000K .......... .......... .......... .......... .......... 92% 25.3M 0s 2024-04-24T16:01:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156050K .......... .......... .......... .......... .......... 92% 42.9M 0s 2024-04-24T16:01:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156100K .......... .......... .......... .......... .......... 92% 25.3M 0s 2024-04-24T16:01:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156150K .......... .......... .......... .......... .......... 92% 36.2M 0s 2024-04-24T16:01:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156200K .......... .......... .......... .......... .......... 92% 24.3M 0s 2024-04-24T16:01:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156250K .......... .......... .......... .......... .......... 92% 49.6M 0s 2024-04-24T16:01:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156300K .......... .......... .......... .......... .......... 92% 27.8M 0s 2024-04-24T16:01:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156350K .......... .......... .......... .......... .......... 92% 23.0M 0s 2024-04-24T16:01:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156400K .......... .......... .......... .......... .......... 92% 30.9M 0s 2024-04-24T16:01:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156450K .......... .......... .......... .......... .......... 92% 24.8M 0s 2024-04-24T16:01:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156500K .......... .......... .......... .......... .......... 92% 24.7M 0s 2024-04-24T16:01:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156550K .......... .......... .......... .......... .......... 92% 52.2M 0s 2024-04-24T16:01:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156600K .......... .......... .......... .......... .......... 92% 36.4M 0s 2024-04-24T16:01:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156650K .......... .......... .......... .......... .......... 92% 23.5M 0s 2024-04-24T16:01:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156700K .......... .......... .......... .......... .......... 92% 34.3M 0s 2024-04-24T16:01:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156750K .......... .......... .......... .......... .......... 93% 19.7M 0s 2024-04-24T16:01:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156800K .......... .......... .......... .......... .......... 93% 22.6M 0s 2024-04-24T16:01:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156850K .......... .......... .......... .......... .......... 93% 33.5M 0s 2024-04-24T16:01:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156900K .......... .......... .......... .......... .......... 93% 37.4M 0s 2024-04-24T16:01:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:156950K .......... .......... .......... .......... .......... 93% 24.4M 0s 2024-04-24T16:01:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157000K .......... .......... .......... .......... .......... 93% 22.6M 0s 2024-04-24T16:01:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157050K .......... .......... .......... .......... .......... 93% 48.4M 0s 2024-04-24T16:01:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157100K .......... .......... .......... .......... .......... 93% 36.2M 0s 2024-04-24T16:01:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157150K .......... .......... .......... .......... .......... 93% 23.8M 0s 2024-04-24T16:01:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157200K .......... .......... .......... .......... .......... 93% 25.4M 0s 2024-04-24T16:01:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157250K .......... .......... .......... .......... .......... 93% 24.5M 0s 2024-04-24T16:01:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157300K .......... .......... .......... .......... .......... 93% 59.6M 0s 2024-04-24T16:01:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157350K .......... .......... .......... .......... .......... 93% 23.1M 0s 2024-04-24T16:01:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157400K .......... .......... .......... .......... .......... 93% 36.2M 0s 2024-04-24T16:01:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157450K .......... .......... .......... .......... .......... 93% 24.4M 0s 2024-04-24T16:01:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157500K .......... .......... .......... .......... .......... 93% 36.6M 0s 2024-04-24T16:01:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157550K .......... .......... .......... .......... .......... 93% 18.0M 0s 2024-04-24T16:01:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157600K .......... .......... .......... .......... .......... 93% 49.3M 0s 2024-04-24T16:01:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157650K .......... .......... .......... .......... .......... 93% 34.2M 0s 2024-04-24T16:01:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157700K .......... .......... .......... .......... .......... 93% 24.8M 0s 2024-04-24T16:01:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157750K .......... .......... .......... .......... .......... 93% 23.8M 0s 2024-04-24T16:01:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157800K .......... .......... .......... .......... .......... 93% 51.3M 0s 2024-04-24T16:01:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157850K .......... .......... .......... .......... .......... 93% 22.7M 0s 2024-04-24T16:01:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157900K .......... .......... .......... .......... .......... 93% 37.2M 0s 2024-04-24T16:01:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:157950K .......... .......... .......... .......... .......... 93% 20.7M 0s 2024-04-24T16:01:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158000K .......... .......... .......... .......... .......... 93% 27.7M 0s 2024-04-24T16:01:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158050K .......... .......... .......... .......... .......... 93% 36.9M 0s 2024-04-24T16:01:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158100K .......... .......... .......... .......... .......... 93% 27.7M 0s 2024-04-24T16:01:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158150K .......... .......... .......... .......... .......... 93% 26.5M 0s 2024-04-24T16:01:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158200K .......... .......... .......... .......... .......... 93% 56.4M 0s 2024-04-24T16:01:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158250K .......... .......... .......... .......... .......... 93% 22.5M 0s 2024-04-24T16:01:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158300K .......... .......... .......... .......... .......... 93% 35.5M 0s 2024-04-24T16:01:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158350K .......... .......... .......... .......... .......... 93% 22.1M 0s 2024-04-24T16:01:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158400K .......... .......... .......... .......... .......... 93% 26.0M 0s 2024-04-24T16:01:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158450K .......... .......... .......... .......... .......... 94% 24.6M 0s 2024-04-24T16:01:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158500K .......... .......... .......... .......... .......... 94% 46.8M 0s 2024-04-24T16:01:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158550K .......... .......... .......... .......... .......... 94% 34.1M 0s 2024-04-24T16:01:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158600K .......... .......... .......... .......... .......... 94% 26.3M 0s 2024-04-24T16:01:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158650K .......... .......... .......... .......... .......... 94% 23.7M 0s 2024-04-24T16:01:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158700K .......... .......... .......... .......... .......... 94% 55.4M 0s 2024-04-24T16:01:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158750K .......... .......... .......... .......... .......... 94% 21.9M 0s 2024-04-24T16:01:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158800K .......... .......... .......... .......... .......... 94% 24.9M 0s 2024-04-24T16:01:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158850K .......... .......... .......... .......... .......... 94% 35.4M 0s 2024-04-24T16:01:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158900K .......... .......... .......... .......... .......... 94% 24.8M 0s 2024-04-24T16:01:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:158950K .......... .......... .......... .......... .......... 94% 21.2M 0s 2024-04-24T16:01:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159000K .......... .......... .......... .......... .......... 94% 48.4M 0s 2024-04-24T16:01:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159050K .......... .......... .......... .......... .......... 94% 24.4M 0s 2024-04-24T16:01:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159100K .......... .......... .......... .......... .......... 94% 50.3M 0s 2024-04-24T16:01:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159150K .......... .......... .......... .......... .......... 94% 23.0M 0s 2024-04-24T16:01:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159200K .......... .......... .......... .......... .......... 94% 29.0M 0s 2024-04-24T16:01:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159250K .......... .......... .......... .......... .......... 94% 35.9M 0s 2024-04-24T16:01:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159300K .......... .......... .......... .......... .......... 94% 25.7M 0s 2024-04-24T16:01:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159350K .......... .......... .......... .......... .......... 94% 27.9M 0s 2024-04-24T16:01:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159400K .......... .......... .......... .......... .......... 94% 37.5M 0s 2024-04-24T16:01:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159450K .......... .......... .......... .......... .......... 94% 20.7M 0s 2024-04-24T16:01:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159500K .......... .......... .......... .......... .......... 94% 53.1M 0s 2024-04-24T16:01:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159550K .......... .......... .......... .......... .......... 94% 20.2M 0s 2024-04-24T16:01:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159600K .......... .......... .......... .......... .......... 94% 34.3M 0s 2024-04-24T16:01:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159650K .......... .......... .......... .......... .......... 94% 25.7M 0s 2024-04-24T16:01:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159700K .......... .......... .......... .......... .......... 94% 35.8M 0s 2024-04-24T16:01:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159750K .......... .......... .......... .......... .......... 94% 23.8M 0s 2024-04-24T16:01:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159800K .......... .......... .......... .......... .......... 94% 38.3M 0s 2024-04-24T16:01:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159850K .......... .......... .......... .......... .......... 94% 29.6M 0s 2024-04-24T16:01:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159900K .......... .......... .......... .......... .......... 94% 38.3M 0s 2024-04-24T16:01:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:159950K .......... .......... .......... .......... .......... 94% 21.1M 0s 2024-04-24T16:01:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160000K .......... .......... .......... .......... .......... 94% 26.2M 0s 2024-04-24T16:01:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160050K .......... .......... .......... .......... .......... 94% 39.7M 0s 2024-04-24T16:01:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160100K .......... .......... .......... .......... .......... 95% 26.5M 0s 2024-04-24T16:01:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160150K .......... .......... .......... .......... .......... 95% 24.5M 0s 2024-04-24T16:01:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160200K .......... .......... .......... .......... .......... 95% 38.4M 0s 2024-04-24T16:01:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160250K .......... .......... .......... .......... .......... 95% 22.5M 0s 2024-04-24T16:01:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160300K .......... .......... .......... .......... .......... 95% 51.4M 0s 2024-04-24T16:01:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160350K .......... .......... .......... .......... .......... 95% 21.6M 0s 2024-04-24T16:01:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160400K .......... .......... .......... .......... .......... 95% 23.8M 0s 2024-04-24T16:01:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160450K .......... .......... .......... .......... .......... 95% 36.8M 0s 2024-04-24T16:01:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160500K .......... .......... .......... .......... .......... 95% 34.0M 0s 2024-04-24T16:01:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160550K .......... .......... .......... .......... .......... 95% 27.2M 0s 2024-04-24T16:01:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160600K .......... .......... .......... .......... .......... 95% 32.5M 0s 2024-04-24T16:01:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160650K .......... .......... .......... .......... .......... 95% 34.7M 0s 2024-04-24T16:01:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160700K .......... .......... .......... .......... .......... 95% 25.2M 0s 2024-04-24T16:01:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160750K .......... .......... .......... .......... .......... 95% 22.6M 0s 2024-04-24T16:01:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160800K .......... .......... .......... .......... .......... 95% 26.7M 0s 2024-04-24T16:01:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160850K .......... .......... .......... .......... .......... 95% 32.8M 0s 2024-04-24T16:01:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160900K .......... .......... .......... .......... .......... 95% 24.5M 0s 2024-04-24T16:01:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:160950K .......... .......... .......... .......... .......... 95% 45.4M 0s 2024-04-24T16:01:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161000K .......... .......... .......... .......... .......... 95% 33.6M 0s 2024-04-24T16:01:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161050K .......... .......... .......... .......... .......... 95% 27.9M 0s 2024-04-24T16:01:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161100K .......... .......... .......... .......... .......... 95% 27.8M 0s 2024-04-24T16:01:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161150K .......... .......... .......... .......... .......... 95% 21.6M 0s 2024-04-24T16:01:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161200K .......... .......... .......... .......... .......... 95% 49.5M 0s 2024-04-24T16:01:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161250K .......... .......... .......... .......... .......... 95% 28.6M 0s 2024-04-24T16:01:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161300K .......... .......... .......... .......... .......... 95% 19.6M 0s 2024-04-24T16:01:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161350K .......... .......... .......... .......... .......... 95% 28.9M 0s 2024-04-24T16:01:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161400K .......... .......... .......... .......... .......... 95% 26.0M 0s 2024-04-24T16:01:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161450K .......... .......... .......... .......... .......... 95% 25.7M 0s 2024-04-24T16:01:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161500K .......... .......... .......... .......... .......... 95% 53.0M 0s 2024-04-24T16:01:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161550K .......... .......... .......... .......... .......... 95% 23.6M 0s 2024-04-24T16:01:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161600K .......... .......... .......... .......... .......... 95% 30.3M 0s 2024-04-24T16:01:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161650K .......... .......... .......... .......... .......... 95% 28.9M 0s 2024-04-24T16:01:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161700K .......... .......... .......... .......... .......... 95% 33.5M 0s 2024-04-24T16:01:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161750K .......... .......... .......... .......... .......... 95% 29.0M 0s 2024-04-24T16:01:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161800K .......... .......... .......... .......... .......... 96% 25.3M 0s 2024-04-24T16:01:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161850K .......... .......... .......... .......... .......... 96% 28.2M 0s 2024-04-24T16:01:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161900K .......... .......... .......... .......... .......... 96% 28.4M 0s 2024-04-24T16:01:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:161950K .......... .......... .......... .......... .......... 96% 27.2M 0s 2024-04-24T16:01:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162000K .......... .......... .......... .......... .......... 96% 37.1M 0s 2024-04-24T16:01:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162050K .......... .......... .......... .......... .......... 96% 26.7M 0s 2024-04-24T16:01:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162100K .......... .......... .......... .......... .......... 96% 25.5M 0s 2024-04-24T16:01:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162150K .......... .......... .......... .......... .......... 96% 36.2M 0s 2024-04-24T16:01:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162200K .......... .......... .......... .......... .......... 96% 22.0M 0s 2024-04-24T16:01:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162250K .......... .......... .......... .......... .......... 96% 55.2M 0s 2024-04-24T16:01:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162300K .......... .......... .......... .......... .......... 96% 23.8M 0s 2024-04-24T16:01:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162350K .......... .......... .......... .......... .......... 96% 21.6M 0s 2024-04-24T16:01:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162400K .......... .......... .......... .......... .......... 96% 37.7M 0s 2024-04-24T16:01:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162450K .......... .......... .......... .......... .......... 96% 28.0M 0s 2024-04-24T16:01:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162500K .......... .......... .......... .......... .......... 96% 32.1M 0s 2024-04-24T16:01:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162550K .......... .......... .......... .......... .......... 96% 22.4M 0s 2024-04-24T16:01:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162600K .......... .......... .......... .......... .......... 96% 39.2M 0s 2024-04-24T16:01:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162650K .......... .......... .......... .......... .......... 96% 26.0M 0s 2024-04-24T16:01:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162700K .......... .......... .......... .......... .......... 96% 28.5M 0s 2024-04-24T16:01:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162750K .......... .......... .......... .......... .......... 96% 29.8M 0s 2024-04-24T16:01:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162800K .......... .......... .......... .......... .......... 96% 32.2M 0s 2024-04-24T16:01:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162850K .......... .......... .......... .......... .......... 96% 31.0M 0s 2024-04-24T16:01:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162900K .......... .......... .......... .......... .......... 96% 29.1M 0s 2024-04-24T16:01:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:162950K .......... .......... .......... .......... .......... 96% 23.5M 0s 2024-04-24T16:01:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163000K .......... .......... .......... .......... .......... 96% 39.6M 0s 2024-04-24T16:01:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163050K .......... .......... .......... .......... .......... 96% 36.2M 0s 2024-04-24T16:01:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163100K .......... .......... .......... .......... .......... 96% 22.9M 0s 2024-04-24T16:01:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163150K .......... .......... .......... .......... .......... 96% 21.7M 0s 2024-04-24T16:01:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163200K .......... .......... .......... .......... .......... 96% 35.9M 0s 2024-04-24T16:01:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163250K .......... .......... .......... .......... .......... 96% 25.3M 0s 2024-04-24T16:01:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163300K .......... .......... .......... .......... .......... 96% 25.0M 0s 2024-04-24T16:01:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163350K .......... .......... .......... .......... .......... 96% 52.0M 0s 2024-04-24T16:01:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163400K .......... .......... .......... .......... .......... 96% 29.6M 0s 2024-04-24T16:01:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163450K .......... .......... .......... .......... .......... 96% 25.9M 0s 2024-04-24T16:01:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163500K .......... .......... .......... .......... .......... 97% 25.5M 0s 2024-04-24T16:01:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163550K .......... .......... .......... .......... .......... 97% 22.6M 0s 2024-04-24T16:01:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163600K .......... .......... .......... .......... .......... 97% 50.4M 0s 2024-04-24T16:01:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163650K .......... .......... .......... .......... .......... 97% 23.2M 0s 2024-04-24T16:01:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163700K .......... .......... .......... .......... .......... 97% 36.0M 0s 2024-04-24T16:01:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163750K .......... .......... .......... .......... .......... 97% 25.4M 0s 2024-04-24T16:01:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163800K .......... .......... .......... .......... .......... 97% 23.8M 0s 2024-04-24T16:01:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163850K .......... .......... .......... .......... .......... 97% 46.3M 0s 2024-04-24T16:01:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163900K .......... .......... .......... .......... .......... 97% 32.2M 0s 2024-04-24T16:01:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:163950K .......... .......... .......... .......... .......... 97% 23.3M 0s 2024-04-24T16:01:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164000K .......... .......... .......... .......... .......... 97% 26.2M 0s 2024-04-24T16:01:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164050K .......... .......... .......... .......... .......... 97% 23.6M 0s 2024-04-24T16:01:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164100K .......... .......... .......... .......... .......... 97% 46.4M 0s 2024-04-24T16:01:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164150K .......... .......... .......... .......... .......... 97% 32.1M 0s 2024-04-24T16:01:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164200K .......... .......... .......... .......... .......... 97% 26.0M 0s 2024-04-24T16:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164250K .......... .......... .......... .......... .......... 97% 23.9M 0s 2024-04-24T16:01:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164300K .......... .......... .......... .......... .......... 97% 50.7M 0s 2024-04-24T16:01:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164350K .......... .......... .......... .......... .......... 97% 21.3M 0s 2024-04-24T16:01:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164400K .......... .......... .......... .......... .......... 97% 34.1M 0s 2024-04-24T16:01:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164450K .......... .......... .......... .......... .......... 97% 27.8M 0s 2024-04-24T16:01:17.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164500K .......... .......... .......... .......... .......... 97% 25.4M 0s 2024-04-24T16:01:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164550K .......... .......... .......... .......... .......... 97% 23.6M 0s 2024-04-24T16:01:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164600K .......... .......... .......... .......... .......... 97% 42.1M 0s 2024-04-24T16:01:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164650K .......... .......... .......... .......... .......... 97% 26.1M 0s 2024-04-24T16:01:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164700K .......... .......... .......... .......... .......... 97% 37.5M 0s 2024-04-24T16:01:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164750K .......... .......... .......... .......... .......... 97% 17.8M 0s 2024-04-24T16:01:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164800K .......... .......... .......... .......... .......... 97% 31.1M 0s 2024-04-24T16:01:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164850K .......... .......... .......... .......... .......... 97% 50.0M 0s 2024-04-24T16:01:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164900K .......... .......... .......... .......... .......... 97% 27.6M 0s 2024-04-24T16:01:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:164950K .......... .......... .......... .......... .......... 97% 37.5M 0s 2024-04-24T16:01:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165000K .......... .......... .......... .......... .......... 97% 31.5M 0s 2024-04-24T16:01:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165050K .......... .......... .......... .......... .......... 97% 31.1M 0s 2024-04-24T16:01:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165100K .......... .......... .......... .......... .......... 97% 25.1M 0s 2024-04-24T16:01:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165150K .......... .......... .......... .......... .......... 98% 24.9M 0s 2024-04-24T16:01:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165200K .......... .......... .......... .......... .......... 98% 22.9M 0s 2024-04-24T16:01:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165250K .......... .......... .......... .......... .......... 98% 32.4M 0s 2024-04-24T16:01:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165300K .......... .......... .......... .......... .......... 98% 35.2M 0s 2024-04-24T16:01:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165350K .......... .......... .......... .......... .......... 98% 29.6M 0s 2024-04-24T16:01:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165400K .......... .......... .......... .......... .......... 98% 23.5M 0s 2024-04-24T16:01:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165450K .......... .......... .......... .......... .......... 98% 36.9M 0s 2024-04-24T16:01:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165500K .......... .......... .......... .......... .......... 98% 26.4M 0s 2024-04-24T16:01:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165550K .......... .......... .......... .......... .......... 98% 21.7M 0s 2024-04-24T16:01:17.760 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165600K .......... .......... .......... .......... .......... 98% 44.8M 0s 2024-04-24T16:01:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165650K .......... .......... .......... .......... .......... 98% 25.2M 0s 2024-04-24T16:01:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165700K .......... .......... .......... .......... .......... 98% 26.9M 0s 2024-04-24T16:01:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165750K .......... .......... .......... .......... .......... 98% 35.5M 0s 2024-04-24T16:01:17.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165800K .......... .......... .......... .......... .......... 98% 23.2M 0s 2024-04-24T16:01:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165850K .......... .......... .......... .......... .......... 98% 35.6M 0s 2024-04-24T16:01:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165900K .......... .......... .......... .......... .......... 98% 30.8M 0s 2024-04-24T16:01:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:165950K .......... .......... .......... .......... .......... 98% 21.4M 0s 2024-04-24T16:01:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166000K .......... .......... .......... .......... .......... 98% 49.9M 0s 2024-04-24T16:01:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166050K .......... .......... .......... .......... .......... 98% 27.4M 0s 2024-04-24T16:01:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166100K .......... .......... .......... .......... .......... 98% 30.3M 0s 2024-04-24T16:01:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166150K .......... .......... .......... .......... .......... 98% 25.0M 0s 2024-04-24T16:01:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166200K .......... .......... .......... .......... .......... 98% 25.6M 0s 2024-04-24T16:01:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166250K .......... .......... .......... .......... .......... 98% 45.6M 0s 2024-04-24T16:01:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166300K .......... .......... .......... .......... .......... 98% 32.2M 0s 2024-04-24T16:01:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166350K .......... .......... .......... .......... .......... 98% 18.4M 0s 2024-04-24T16:01:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166400K .......... .......... .......... .......... .......... 98% 36.3M 0s 2024-04-24T16:01:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166450K .......... .......... .......... .......... .......... 98% 23.3M 0s 2024-04-24T16:01:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166500K .......... .......... .......... .......... .......... 98% 40.0M 0s 2024-04-24T16:01:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166550K .......... .......... .......... .......... .......... 98% 20.5M 0s 2024-04-24T16:01:17.794 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166600K .......... .......... .......... .......... .......... 98% 54.7M 0s 2024-04-24T16:01:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166650K .......... .......... .......... .......... .......... 98% 24.7M 0s 2024-04-24T16:01:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166700K .......... .......... .......... .......... .......... 98% 40.6M 0s 2024-04-24T16:01:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166750K .......... .......... .......... .......... .......... 98% 24.8M 0s 2024-04-24T16:01:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166800K .......... .......... .......... .......... .......... 98% 26.5M 0s 2024-04-24T16:01:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166850K .......... .......... .......... .......... .......... 99% 25.5M 0s 2024-04-24T16:01:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166900K .......... .......... .......... .......... .......... 99% 31.1M 0s 2024-04-24T16:01:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:166950K .......... .......... .......... .......... .......... 99% 31.5M 0s 2024-04-24T16:01:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167000K .......... .......... .......... .......... .......... 99% 35.3M 0s 2024-04-24T16:01:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167050K .......... .......... .......... .......... .......... 99% 31.4M 0s 2024-04-24T16:01:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167100K .......... .......... .......... .......... .......... 99% 20.1M 0s 2024-04-24T16:01:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167150K .......... .......... .......... .......... .......... 99% 29.3M 0s 2024-04-24T16:01:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167200K .......... .......... .......... .......... .......... 99% 39.2M 0s 2024-04-24T16:01:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167250K .......... .......... .......... .......... .......... 99% 24.8M 0s 2024-04-24T16:01:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167300K .......... .......... .......... .......... .......... 99% 19.5M 0s 2024-04-24T16:01:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167350K .......... .......... .......... .......... .......... 99% 29.7M 0s 2024-04-24T16:01:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167400K .......... .......... .......... .......... .......... 99% 26.4M 0s 2024-04-24T16:01:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167450K .......... .......... .......... .......... .......... 99% 25.1M 0s 2024-04-24T16:01:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167500K .......... .......... .......... .......... .......... 99% 47.9M 0s 2024-04-24T16:01:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167550K .......... .......... .......... .......... .......... 99% 23.4M 0s 2024-04-24T16:01:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167600K .......... .......... .......... .......... .......... 99% 23.8M 0s 2024-04-24T16:01:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167650K .......... .......... .......... .......... .......... 99% 31.0M 0s 2024-04-24T16:01:17.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167700K .......... .......... .......... .......... .......... 99% 42.3M 0s 2024-04-24T16:01:17.834 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167750K .......... .......... .......... .......... .......... 99% 34.4M 0s 2024-04-24T16:01:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167800K .......... .......... .......... .......... .......... 99% 23.0M 0s 2024-04-24T16:01:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167850K .......... .......... .......... .......... .......... 99% 29.0M 0s 2024-04-24T16:01:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167900K .......... .......... .......... .......... .......... 99% 34.0M 0s 2024-04-24T16:01:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:167950K .......... .......... .......... .......... .......... 99% 26.2M 0s 2024-04-24T16:01:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168000K .......... .......... .......... .......... .......... 99% 27.8M 0s 2024-04-24T16:01:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168050K .......... .......... .......... .......... .......... 99% 31.9M 0s 2024-04-24T16:01:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168100K .......... .......... .......... .......... .......... 99% 27.6M 0s 2024-04-24T16:01:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168150K .......... .......... .......... .......... .......... 99% 34.2M 0s 2024-04-24T16:01:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168200K .......... .......... .......... .......... .......... 99% 33.3M 0s 2024-04-24T16:01:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168250K .......... .......... .......... .......... .......... 99% 23.8M 0s 2024-04-24T16:01:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168300K .......... .......... .......... .......... .......... 99% 35.5M 0s 2024-04-24T16:01:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168350K .......... .......... .......... .......... .......... 99% 17.5M 0s 2024-04-24T16:01:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168400K .......... .......... .......... .......... .......... 99% 47.2M 0s 2024-04-24T16:01:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168450K .......... .......... .......... .......... .......... 99% 25.2M 0s 2024-04-24T16:01:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168500K .......... .......... .......... .......... .......... 99% 33.7M 0s 2024-04-24T16:01:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:168550K .......... .......... . 100% 266M=5.9s 2024-04-24T16:01:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2024-04-24T16:01:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:2024-04-24 16:01:17 (28.0 MB/s) - ‘linux-5.4.tar.gz’ saved [172616875/172616875] 2024-04-24T16:01:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: 2024-04-24T16:01:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:01:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:01:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:01:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:01:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/lost+found 2024-04-24T16:01:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=46290 2024-04-24T16:01:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:01:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:01:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T16:01:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:01:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:01:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:01:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:01:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:01:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:01:18.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 5 2024-04-24T16:01:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:01:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:01:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:01:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:01:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 6 2024-04-24T16:01:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:01:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:01:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:01:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:01:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 8 2024-04-24T16:01:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:01:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:01:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:01:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:01:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 10 2024-04-24T16:01:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:01:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:01:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:01:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:01:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 12 2024-04-24T16:01:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:01:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:01:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:01:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:01:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 14 2024-04-24T16:01:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:01:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:01:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:01:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:01:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 16 2024-04-24T16:01:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:01:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:01:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:01:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:01:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 18 2024-04-24T16:01:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:01:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:01:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:01:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:01:56.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:01:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:01:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:01:58.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 20 2024-04-24T16:01:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 22 2024-04-24T16:02:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:06.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 24 2024-04-24T16:02:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 26 2024-04-24T16:02:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 28 2024-04-24T16:02:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 30 2024-04-24T16:02:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 32 2024-04-24T16:02:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 34 2024-04-24T16:02:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:38.342 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 36 2024-04-24T16:02:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:43.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 38 2024-04-24T16:02:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 40 2024-04-24T16:02:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 42 2024-04-24T16:02:53.378 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:02:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:02:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:02:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:02:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:02:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:02:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:02:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 44 2024-04-24T16:02:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:03:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:03:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:03:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:03:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:03:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:03:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:03:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 46 2024-04-24T16:03:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:03:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:03:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:03:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:03:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:03:06.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:03:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:03:08.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 48 2024-04-24T16:03:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:03:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:03:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:03:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:03:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:03:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:03:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:03:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 50 2024-04-24T16:03:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:03:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:03:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:03:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:03:16.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:03:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:03:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:03:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 52 2024-04-24T16:03:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:03:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:03:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 54 2024-04-24T16:03:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:03:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:03:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:03:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:03:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:03:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:03:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:03:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 56 2024-04-24T16:03:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:03:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:03:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:03:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:03:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:03:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:03:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:03:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 58 2024-04-24T16:03:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:03:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:03:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:03:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:03:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:03:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:03:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:03:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 60 2024-04-24T16:03:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:03:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:03:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:03:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:03:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:03:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:03:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:03:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 62 2024-04-24T16:03:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:06:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T16:06:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T16:06:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 46290 2024-04-24T16:06:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T16:06:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:06:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:06:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=faf04d048bc15afbb6f52400502787e7 2024-04-24T16:06:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T16:06:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T16:06:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:06:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:06:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:06:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T16:06:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T16:06:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T16:06:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:06:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:06:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:06:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:06:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:06:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:06:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:06:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:06:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:06:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:06:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:06:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:06:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:06:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:06:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:06:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:06:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:06:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:06:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:06:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:06:39.450 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+stopped 2024-04-24T16:06:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: local image is primary 2024-04-24T16:06:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:06:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:06:35 2024-04-24T16:06:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:06:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:06:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:06:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T16:06:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:06:38' 2024-04-24T16:06:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:06:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 1 2024-04-24T16:06:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:06:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 2 2024-04-24T16:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:06:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:06:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:06:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:06:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:06:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:06:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:06:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:06:40 2024-04-24T16:06:41.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:06:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:06:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:06:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:06:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:06:40' 2024-04-24T16:06:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:06:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:06:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:06:41.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:06:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:06:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:06:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:06:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:06:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:06:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:06:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:06:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:06:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:06:40' 2024-04-24T16:06:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 16:06:40' 2024-04-24T16:06:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:06:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:06:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:06:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:06:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:06:40' 2024-04-24T16:06:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:06:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:06:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T16:06:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 16:06:40cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 16:06:40' 2024-04-24T16:06:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:06:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:06:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:06:40' 2024-04-24T16:06:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:06:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:06:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:06:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:06:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:06:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:06:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:06:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T16:06:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:06:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:06:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:06:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:06:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:06:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:06:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:06:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:06:42.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:06:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:06:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:06:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:06:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:06:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:06:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:06:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:06:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:06:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:06:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:06:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:06:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:06:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:06:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:06:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:06:40 2024-04-24T16:06:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:06:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:06:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:06:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:06:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:06:40' 2024-04-24T16:06:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:06:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:06:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:06:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:06:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:06:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:06:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:06:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:06:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:06:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:06:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:06:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:06:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:06:40' 2024-04-24T16:06:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 16:06:40' 2024-04-24T16:06:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:06:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:06:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:06:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:06:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:06:40' 2024-04-24T16:06:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:06:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:06:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T16:06:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 16:06:40cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 16:06:40' 2024-04-24T16:06:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:06:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:06:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:06:40' 2024-04-24T16:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:06:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:06:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:06:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:06:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:06:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:06:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:06:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T16:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T16:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T16:06:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T16:06:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:06:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:06:43.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:06:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:06:43.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:07:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=faf04d048bc15afbb6f52400502787e7 2024-04-24T16:07:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ faf04d048bc15afbb6f52400502787e7 != \f\a\f\0\4\d\0\4\8\b\c\1\5\a\f\b\b\6\f\5\2\4\0\0\5\0\2\7\8\7\e\7 ]] 2024-04-24T16:07:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T16:07:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T16:07:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T16:07:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:07:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:07:02.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:07:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:07:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T16:07:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=47284 2024-04-24T16:07:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:07:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:07:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T16:07:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:07:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:07:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:07:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 69 2024-04-24T16:07:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 71 2024-04-24T16:07:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 73 2024-04-24T16:07:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 75 2024-04-24T16:07:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 77 2024-04-24T16:07:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 79 2024-04-24T16:07:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 81 2024-04-24T16:07:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 83 2024-04-24T16:07:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 85 2024-04-24T16:07:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 87 2024-04-24T16:07:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 89 2024-04-24T16:07:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:07:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 91 2024-04-24T16:07:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:07:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:07:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:07:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:07:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:07:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:07:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 93 2024-04-24T16:08:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 95 2024-04-24T16:08:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 97 2024-04-24T16:08:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 99 2024-04-24T16:08:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 101 2024-04-24T16:08:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 103 2024-04-24T16:08:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 105 2024-04-24T16:08:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 107 2024-04-24T16:08:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:41.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 109 2024-04-24T16:08:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 111 2024-04-24T16:08:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 113 2024-04-24T16:08:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:08:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 115 2024-04-24T16:08:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:08:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:08:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:08:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:08:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:08:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:08:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:09:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 117 2024-04-24T16:09:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:09:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:09:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:09:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:09:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:09:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:09:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:09:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 119 2024-04-24T16:09:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:09:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:09:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:09:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:09:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:09:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:09:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:09:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 121 2024-04-24T16:09:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:09:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:09:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:09:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:09:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:09:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:09:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:09:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 123 2024-04-24T16:09:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:09:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:09:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:09:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:09:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:09:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:09:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:09:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 125 2024-04-24T16:09:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:09:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:09:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:09:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:09:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:09:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:09:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:09:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 127 2024-04-24T16:09:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:12:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T16:12:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T16:12:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 47284 2024-04-24T16:12:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T16:12:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:12:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:12:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=6982640cb9a4b28195daf4bffa6c038e 2024-04-24T16:12:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T16:12:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T16:12:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:12:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T16:12:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T16:12:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T16:12:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:12:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:12:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:12:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:12:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:12:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:12:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:12:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:12:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:12:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:12:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:12:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:12:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:12:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:12:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:12:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:12:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+stopped 2024-04-24T16:12:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: local image is primary 2024-04-24T16:12:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:12:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:12:20 2024-04-24T16:12:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:12:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:12:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:12:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T16:12:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:12:24' 2024-04-24T16:12:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:12:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 1 2024-04-24T16:12:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:12:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 2 2024-04-24T16:12:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:12:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:12:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:12:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:12:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:12:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:12:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:12:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:12:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:12:25 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:12:25' 2024-04-24T16:12:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:12:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:12:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:12:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:12:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:12:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:12:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:12:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:12:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:12:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:12:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:12:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:12:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:12:25' 2024-04-24T16:12:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 16:12:25' 2024-04-24T16:12:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:12:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:12:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:12:25' 2024-04-24T16:12:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:12:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:12:27.057 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T16:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 16:12:25cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 16:12:25' 2024-04-24T16:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:12:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:12:25' 2024-04-24T16:12:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:12:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:12:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:12:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:12:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:12:27.110 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:12:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:12:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T16:12:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:12:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:12:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:12:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:12:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:12:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:12:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:12:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:12:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:12:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:12:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:12:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:12:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:12:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:12:25 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:12:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:12:25' 2024-04-24T16:12:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:12:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:12:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:12:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:12:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:12:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:12:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:12:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:12:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:12:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:12:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:12:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:12:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:12:25' 2024-04-24T16:12:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 16:12:25' 2024-04-24T16:12:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:12:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:12:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:12:25' 2024-04-24T16:12:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:12:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:12:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T16:12:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 16:12:25cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 16:12:25' 2024-04-24T16:12:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:12:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:12:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:12:25' 2024-04-24T16:12:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:12:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:12:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:12:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:12:28.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:12:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:12:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:12:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T16:12:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:12:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T16:12:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T16:12:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T16:12:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:12:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:12:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:12:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:12:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:12:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=6982640cb9a4b28195daf4bffa6c038e 2024-04-24T16:12:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 6982640cb9a4b28195daf4bffa6c038e != \6\9\8\2\6\4\0\c\b\9\a\4\b\2\8\1\9\5\d\a\f\4\b\f\f\a\6\c\0\3\8\e ]] 2024-04-24T16:12:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T16:12:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T16:12:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T16:12:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:12:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:12:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:12:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:12:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T16:12:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=48468 2024-04-24T16:12:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:12:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T16:12:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:12:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:12:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:12:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:12:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:12:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:12:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:12:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:12:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:12:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 131 2024-04-24T16:12:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:12:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:12:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:12:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:12:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:12:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 133 2024-04-24T16:12:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:12:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:12:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:12:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:12:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:12:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:12:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:01.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 135 2024-04-24T16:13:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 137 2024-04-24T16:13:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 139 2024-04-24T16:13:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 141 2024-04-24T16:13:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:19.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:19.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 143 2024-04-24T16:13:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 145 2024-04-24T16:13:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:29.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:31.281 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 147 2024-04-24T16:13:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 149 2024-04-24T16:13:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 151 2024-04-24T16:13:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 153 2024-04-24T16:13:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 155 2024-04-24T16:13:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:13:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 157 2024-04-24T16:13:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:13:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:13:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:13:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:13:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:13:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:13:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 159 2024-04-24T16:14:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 161 2024-04-24T16:14:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 163 2024-04-24T16:14:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 165 2024-04-24T16:14:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 167 2024-04-24T16:14:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 169 2024-04-24T16:14:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 171 2024-04-24T16:14:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 173 2024-04-24T16:14:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 175 2024-04-24T16:14:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 177 2024-04-24T16:14:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 179 2024-04-24T16:14:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:14:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 181 2024-04-24T16:14:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:14:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:14:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:14:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:14:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:14:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:14:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:15:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 183 2024-04-24T16:15:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:15:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:15:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:15:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:15:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:15:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:15:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:15:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 185 2024-04-24T16:15:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:15:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:15:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:15:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:15:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:15:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:15:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:15:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 187 2024-04-24T16:15:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:15:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:15:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:15:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:15:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:15:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:15:14.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:15:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 189 2024-04-24T16:15:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:17:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T16:17:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T16:17:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 48468 2024-04-24T16:17:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T16:17:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:17:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:18:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=9355a2dd28690ff53c3c713f57a571d8 2024-04-24T16:18:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T16:18:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T16:18:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:18:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T16:18:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T16:18:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T16:18:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:18:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:18:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:18:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:18:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:18:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:18:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:18:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:18:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:18:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:18:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:18:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:18:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:18:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:18:10 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:18:10' 2024-04-24T16:18:13.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:18:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:18:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:18:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:18:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:18:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:18:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:18:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:18:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:18:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:18:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:18:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:18:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:18:10' 2024-04-24T16:18:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 16:18:10' 2024-04-24T16:18:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:18:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:18:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:18:10' 2024-04-24T16:18:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:18:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:18:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T16:18:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 16:18:10cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 16:18:10' 2024-04-24T16:18:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:18:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:18:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:18:10' 2024-04-24T16:18:13.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:18:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:18:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:18:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:18:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T16:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:18:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:18:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:18:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:18:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:18:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:18:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:18:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:18:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:18:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:18:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:18:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:18:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:18:10 2024-04-24T16:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:18:10' 2024-04-24T16:18:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:18:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:18:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:18:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:18:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:18:14.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:18:10' 2024-04-24T16:18:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 16:18:10' 2024-04-24T16:18:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:18:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:18:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:18:10' 2024-04-24T16:18:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:18:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:18:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T16:18:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 16:18:10cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 16:18:10' 2024-04-24T16:18:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:18:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:18:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:18:10' 2024-04-24T16:18:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:18:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:18:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:18:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:18:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:18:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:18:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:18:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T16:18:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:18:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T16:18:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T16:18:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T16:18:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:18:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:18:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:18:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:18:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:18:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=9355a2dd28690ff53c3c713f57a571d8 2024-04-24T16:18:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 9355a2dd28690ff53c3c713f57a571d8 != \9\3\5\5\a\2\d\d\2\8\6\9\0\f\f\5\3\c\3\c\7\1\3\f\5\7\a\5\7\1\d\8 ]] 2024-04-24T16:18:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T16:18:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T16:18:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T16:18:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:18:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:18:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:18:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:18:35.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T16:18:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=49432 2024-04-24T16:18:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:18:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T16:18:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:18:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:18:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:18:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:18:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:18:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:18:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:18:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:18:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:18:37.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 196 2024-04-24T16:18:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:18:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:18:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:18:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:18:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:18:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 198 2024-04-24T16:18:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:18:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 200 2024-04-24T16:18:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:18:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:18:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:18:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:18:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:18:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 202 2024-04-24T16:18:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:18:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:18:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:18:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:18:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:18:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:18:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:18:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 204 2024-04-24T16:18:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 206 2024-04-24T16:19:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 208 2024-04-24T16:19:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 210 2024-04-24T16:19:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 212 2024-04-24T16:19:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 214 2024-04-24T16:19:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 216 2024-04-24T16:19:26.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 218 2024-04-24T16:19:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 220 2024-04-24T16:19:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 222 2024-04-24T16:19:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 224 2024-04-24T16:19:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 226 2024-04-24T16:19:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:19:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 228 2024-04-24T16:19:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:19:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:19:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:19:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:19:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:19:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:19:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 230 2024-04-24T16:20:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 232 2024-04-24T16:20:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 234 2024-04-24T16:20:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 236 2024-04-24T16:20:16.974 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 238 2024-04-24T16:20:21.984 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 240 2024-04-24T16:20:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 242 2024-04-24T16:20:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 244 2024-04-24T16:20:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 246 2024-04-24T16:20:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 248 2024-04-24T16:20:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 250 2024-04-24T16:20:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:20:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 252 2024-04-24T16:20:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:20:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:20:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:20:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:20:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:20:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:20:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:21:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 254 2024-04-24T16:21:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:23:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T16:23:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T16:23:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 49432 2024-04-24T16:23:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T16:23:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:23:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:23:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=d9bc7154a204dcef0ed6799c7c0e34fc 2024-04-24T16:23:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T16:23:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T16:23:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:23:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:23:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:23:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T16:23:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T16:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T16:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:23:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:24:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:24:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:24:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+stopped 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: local image is primary 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:23:55 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T16:24:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:23:57' 2024-04-24T16:24:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:24:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 1 2024-04-24T16:24:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:24:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 2 2024-04-24T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:24:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:24:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:24:00 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:24:00' 2024-04-24T16:24:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:24:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:24:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:24:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:24:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:24:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:24:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:24:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:24:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:24:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:24:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:24:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:24:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:24:00' 2024-04-24T16:24:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 16:24:00' 2024-04-24T16:24:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:24:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:24:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:24:00' 2024-04-24T16:24:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:24:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:24:02.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T16:24:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 16:24:00cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 16:24:00' 2024-04-24T16:24:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:24:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:24:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:24:00' 2024-04-24T16:24:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:24:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:24:02.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:24:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:24:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:24:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T16:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:24:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:24:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:24:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:24:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:24:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:24:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:24:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:24:03.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:24:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:24:03.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:24:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:24:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:24:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:24:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:24:00 2024-04-24T16:24:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:24:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:24:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:24:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:24:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:24:00' 2024-04-24T16:24:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:24:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:24:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:24:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:24:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:24:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:24:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:24:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:24:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:24:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:24:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:24:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:24:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:24:00' 2024-04-24T16:24:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 16:24:00' 2024-04-24T16:24:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:24:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:24:00' 2024-04-24T16:24:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:24:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:24:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T16:24:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 16:24:00cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 16:24:00' 2024-04-24T16:24:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:24:03.316 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:24:03.317 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:24:00' 2024-04-24T16:24:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:24:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:24:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:24:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:24:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:24:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:24:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:24:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T16:24:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T16:24:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T16:24:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T16:24:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:24:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:24:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:24:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:24:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:24:05.931 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T16:24:05.933+0000 7f7d32b02700 -1 reset not still connected to 0x558c920cfd40 2024-04-24T16:24:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=d9bc7154a204dcef0ed6799c7c0e34fc 2024-04-24T16:24:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ d9bc7154a204dcef0ed6799c7c0e34fc != \d\9\b\c\7\1\5\4\a\2\0\4\d\c\e\f\0\e\d\6\7\9\9\c\7\c\0\e\3\4\f\c ]] 2024-04-24T16:24:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T16:24:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T16:24:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T16:24:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:24:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:24:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:24:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:24:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=50444 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:24:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:24:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:24:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:24:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 259 2024-04-24T16:24:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:24:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:24:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:24:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:24:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 261 2024-04-24T16:24:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:24:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:24:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:24:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:33.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:24:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 263 2024-04-24T16:24:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:24:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:24:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:24:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:24:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 265 2024-04-24T16:24:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:24:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 267 2024-04-24T16:24:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:24:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:24:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:24:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:24:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 269 2024-04-24T16:24:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:24:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:24:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:24:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:52.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:24:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 271 2024-04-24T16:24:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:24:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:24:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 273 2024-04-24T16:24:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 275 2024-04-24T16:25:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 277 2024-04-24T16:25:09.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 279 2024-04-24T16:25:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 281 2024-04-24T16:25:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 283 2024-04-24T16:25:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 285 2024-04-24T16:25:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:32.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 287 2024-04-24T16:25:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 289 2024-04-24T16:25:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 291 2024-04-24T16:25:44.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 293 2024-04-24T16:25:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 295 2024-04-24T16:25:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:25:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:25:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:25:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:25:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:25:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:25:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:25:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 297 2024-04-24T16:25:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:26:01.373 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:26:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:26:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:26:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:26:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:26:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:26:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 299 2024-04-24T16:26:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:26:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:26:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:26:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:26:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:26:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:26:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:26:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 301 2024-04-24T16:26:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:26:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:26:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:26:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:26:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:26:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:26:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:26:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 303 2024-04-24T16:26:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:26:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:26:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:26:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:26:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:26:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:26:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:26:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 305 2024-04-24T16:26:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:26:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:26:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:26:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:26:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:26:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:26:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:26:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 307 2024-04-24T16:26:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:26:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:26:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:26:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:26:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:26:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:26:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:26:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 309 2024-04-24T16:26:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:26:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:26:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:26:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:26:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:26:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:26:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:26:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 311 2024-04-24T16:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:26:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:26:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:26:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:26:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:26:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:26:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:26:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 313 2024-04-24T16:26:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:26:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:26:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:26:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:26:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:26:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:26:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:26:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 315 2024-04-24T16:26:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:26:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:26:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:26:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:26:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:26:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:26:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:26:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 317 2024-04-24T16:26:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:29:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T16:29:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T16:29:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 50444 2024-04-24T16:29:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T16:29:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:29:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:29:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=353453f4a799c129f906ae27278aebe7 2024-04-24T16:29:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T16:29:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T16:29:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:29:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:29:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:29:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T16:29:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T16:29:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T16:29:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:29:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:29:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:29:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:29:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:29:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:29:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:29:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:29:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:29:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:29:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:29:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:29:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:29:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:29:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:29:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:29:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:29:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:29:45 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:29:45' 2024-04-24T16:29:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:29:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:29:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:29:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:29:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:29:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:29:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:29:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:29:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:29:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:29:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:29:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:29:45' 2024-04-24T16:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 16:29:45' 2024-04-24T16:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:29:45' 2024-04-24T16:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:29:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T16:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 16:29:45cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 16:29:45' 2024-04-24T16:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:29:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:29:45' 2024-04-24T16:29:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:29:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:29:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:29:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:29:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:29:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:29:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:29:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:29:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:29:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:29:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:29:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:29:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:29:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:29:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:29:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:29:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:29:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:29:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:29:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:29:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:29:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:29:45 2024-04-24T16:29:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:29:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:29:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:29:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:29:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:29:45' 2024-04-24T16:29:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:29:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:29:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:29:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:29:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:29:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:29:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:29:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:29:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:29:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:29:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:29:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:29:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:29:45' 2024-04-24T16:29:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 16:29:45' 2024-04-24T16:29:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:29:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:29:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:29:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:29:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:29:45' 2024-04-24T16:29:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:29:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:29:48.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T16:29:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 16:29:45cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 16:29:45' 2024-04-24T16:29:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:29:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:29:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:29:45' 2024-04-24T16:29:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:29:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:29:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:29:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:29:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:29:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:29:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:29:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T16:29:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:29:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:29:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:29:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T16:29:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T16:29:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T16:29:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:29:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:29:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:29:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:29:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:30:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=353453f4a799c129f906ae27278aebe7 2024-04-24T16:30:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 353453f4a799c129f906ae27278aebe7 != \3\5\3\4\5\3\f\4\a\7\9\9\c\1\2\9\f\9\0\6\a\e\2\7\2\7\8\a\e\b\e\7 ]] 2024-04-24T16:30:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T16:30:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T16:30:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T16:30:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:30:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:30:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:30:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:30:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T16:30:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=51414 2024-04-24T16:30:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:30:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T16:30:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:30:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:30:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:30:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:30:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:30:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 324 2024-04-24T16:30:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:30:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:14.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:30:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 326 2024-04-24T16:30:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:30:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:30:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 328 2024-04-24T16:30:21.225 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:30:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:30:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 330 2024-04-24T16:30:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:30:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:30:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 332 2024-04-24T16:30:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:30:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:30:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 334 2024-04-24T16:30:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:30:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:30:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 336 2024-04-24T16:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:30:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:30:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 338 2024-04-24T16:30:46.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:30:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:30:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 340 2024-04-24T16:30:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:30:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:30:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 342 2024-04-24T16:30:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:30:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:30:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:30:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:30:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:30:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:30:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 344 2024-04-24T16:31:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:06.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 346 2024-04-24T16:31:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 348 2024-04-24T16:31:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:14.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 350 2024-04-24T16:31:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 352 2024-04-24T16:31:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 354 2024-04-24T16:31:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:29.150 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:29.150 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 356 2024-04-24T16:31:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:36.167 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 358 2024-04-24T16:31:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:39.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 360 2024-04-24T16:31:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 362 2024-04-24T16:31:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 364 2024-04-24T16:31:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:31:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 366 2024-04-24T16:31:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:31:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:31:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:31:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:31:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:31:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:31:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:32:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 368 2024-04-24T16:32:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:32:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:32:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:32:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:32:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:32:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:32:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:32:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 370 2024-04-24T16:32:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:32:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:32:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:32:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:32:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:32:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:32:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:32:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 372 2024-04-24T16:32:11.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:32:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:32:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:32:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:32:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:32:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:32:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:32:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 374 2024-04-24T16:32:16.145 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:32:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:32:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:32:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:32:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:32:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:32:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:32:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 376 2024-04-24T16:32:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:32:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:32:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:32:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:32:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:32:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:32:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:32:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 378 2024-04-24T16:32:26.178 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:32:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:32:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:32:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:32:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:32:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:32:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:32:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 380 2024-04-24T16:32:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:32:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:32:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:32:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:32:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:32:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:32:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:32:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 382 2024-04-24T16:32:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:35:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T16:35:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T16:35:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 51414 2024-04-24T16:35:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T16:35:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:35:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:35:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=3c1218b4755274f2bd5331f924e0a67b 2024-04-24T16:35:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T16:35:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T16:35:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:35:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:35:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:35:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T16:35:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T16:35:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T16:35:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:35:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:35:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:35:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:35:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:35:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:35:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:35:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:35:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:35:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:35:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:35:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:35:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:35:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:35:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:35:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:35:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:35:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:35:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:35:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:35:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:35:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:35:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:35:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:35:30 2024-04-24T16:35:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:35:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:35:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:35:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:35:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:35:30' 2024-04-24T16:35:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:35:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:35:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:35:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:35:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:35:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:35:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:35:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:35:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:35:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:35:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:35:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:35:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:35:30' 2024-04-24T16:35:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 16:35:30' 2024-04-24T16:35:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:35:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:35:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:35:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:35:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:35:30' 2024-04-24T16:35:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:35:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:35:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T16:35:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 16:35:30cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 16:35:30' 2024-04-24T16:35:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:35:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:35:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:35:30' 2024-04-24T16:35:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:35:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:35:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:35:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:35:33.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:35:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:35:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:35:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:35:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:35:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:35:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:35:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:35:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:35:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:35:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:35:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:35:30 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:35:30' 2024-04-24T16:35:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:35:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:35:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:35:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:35:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:35:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:35:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:35:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:35:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:35:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:35:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:35:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:35:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:35:30' 2024-04-24T16:35:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 16:35:30' 2024-04-24T16:35:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:35:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:35:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:35:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:35:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:35:30' 2024-04-24T16:35:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:35:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:35:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T16:35:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 16:35:30cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 16:35:30' 2024-04-24T16:35:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:35:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:35:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:35:30' 2024-04-24T16:35:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:35:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:35:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:35:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:35:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:35:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:35:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:35:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T16:35:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:35:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:35:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:35:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T16:35:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T16:35:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T16:35:35.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:35:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:35:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:35:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:35:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:35:54.141 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=3c1218b4755274f2bd5331f924e0a67b 2024-04-24T16:35:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 3c1218b4755274f2bd5331f924e0a67b != \3\c\1\2\1\8\b\4\7\5\5\2\7\4\f\2\b\d\5\3\3\1\f\9\2\4\e\0\a\6\7\b ]] 2024-04-24T16:35:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T16:35:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T16:35:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T16:35:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:35:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:35:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:35:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:35:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T16:35:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=52418 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:35:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:35:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:35:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:35:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 387 2024-04-24T16:35:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 389 2024-04-24T16:36:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 391 2024-04-24T16:36:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 393 2024-04-24T16:36:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 395 2024-04-24T16:36:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 397 2024-04-24T16:36:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 399 2024-04-24T16:36:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 401 2024-04-24T16:36:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 403 2024-04-24T16:36:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:40.713 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 405 2024-04-24T16:36:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 407 2024-04-24T16:36:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 409 2024-04-24T16:36:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:36:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:36:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:36:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:36:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:36:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:36:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:36:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 411 2024-04-24T16:36:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 413 2024-04-24T16:37:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:07.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 415 2024-04-24T16:37:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 417 2024-04-24T16:37:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:15.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 419 2024-04-24T16:37:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 421 2024-04-24T16:37:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 423 2024-04-24T16:37:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 425 2024-04-24T16:37:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 427 2024-04-24T16:37:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:40.832 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:42.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 429 2024-04-24T16:37:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 431 2024-04-24T16:37:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 433 2024-04-24T16:37:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:37:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:37:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:37:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:37:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:37:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:37:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:37:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 435 2024-04-24T16:37:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:38:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:38:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 437 2024-04-24T16:38:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:38:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:38:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:38:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:38:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:38:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:38:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:38:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 439 2024-04-24T16:38:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:38:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:38:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:38:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:38:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:38:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:38:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:38:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 441 2024-04-24T16:38:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:38:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:38:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:38:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:38:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:38:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:38:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:38:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 443 2024-04-24T16:38:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:38:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:38:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:38:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:38:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:38:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:38:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:38:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 445 2024-04-24T16:38:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:40:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T16:40:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T16:40:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 52418 2024-04-24T16:40:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T16:40:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:40:56.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:41:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=83c4b998f56088a107d1b3f868536850 2024-04-24T16:41:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T16:41:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T16:41:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:41:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T16:41:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T16:41:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T16:41:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:41:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:41:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:41:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:41:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:41:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:41:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:41:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:41:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:41:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:41:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:41:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:41:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:41:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:41:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:41:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:41:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:41:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:41:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:41:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:41:15 2024-04-24T16:41:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:41:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:41:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:41:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T16:41:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:41:17' 2024-04-24T16:41:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:41:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:41:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:41:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:41:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:41:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:41:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:41:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:41:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:41:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:41:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:41:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:41:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:41:15' 2024-04-24T16:41:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 16:41:15' 2024-04-24T16:41:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:41:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:41:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:41:15' 2024-04-24T16:41:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:41:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:41:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T16:41:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 16:41:15cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image demoted2024-04-24 16:41:17' 2024-04-24T16:41:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:41:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:41:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:41:15' 2024-04-24T16:41:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:41:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:41:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:41:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:41:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:41:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:41:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:41:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:41:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:41:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:41:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:41:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:41:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:41:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:41:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:41:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:41:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:41:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T16:41:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:41:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:41:17 2024-04-24T16:41:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:41:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:41:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:41:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:41:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:41:15' 2024-04-24T16:41:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:41:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:41:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:41:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image demoted 2024-04-24T16:41:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:41:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:41:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:41:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:41:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:41:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:41:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:41:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:41:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:41:17' 2024-04-24T16:41:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 16:41:17' 2024-04-24T16:41:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:41:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:41:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:41:17' 2024-04-24T16:41:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:41:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:41:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T16:41:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image demoted42594265mirror.0smithi1362024-04-24 16:41:17cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 16:41:15' 2024-04-24T16:41:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:41:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:41:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:41:17' 2024-04-24T16:41:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:41:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:41:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:41:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:41:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:41:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:41:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:41:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T16:41:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:41:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T16:41:19.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T16:41:20.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T16:41:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:41:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:41:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:41:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:41:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=83c4b998f56088a107d1b3f868536850 2024-04-24T16:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 83c4b998f56088a107d1b3f868536850 != \8\3\c\4\b\9\9\8\f\5\6\0\8\8\a\1\0\7\d\1\b\3\f\8\6\8\5\3\6\8\5\0 ]] 2024-04-24T16:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T16:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T16:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T16:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:41:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:41:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:41:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T16:41:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=53424 2024-04-24T16:41:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:41:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:41:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:41:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T16:41:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:41:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:41:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:41:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:41:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:41:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:41:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:41:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 452 2024-04-24T16:41:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:41:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:41:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:45.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:41:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 454 2024-04-24T16:41:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:41:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:41:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:41:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:41:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:41:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 456 2024-04-24T16:41:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:41:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:41:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:41:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:41:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:41:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:41:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:41:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 458 2024-04-24T16:41:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 460 2024-04-24T16:42:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 462 2024-04-24T16:42:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 464 2024-04-24T16:42:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:15.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 466 2024-04-24T16:42:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 468 2024-04-24T16:42:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:26.246 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T16:42:26.246+0000 7f7d32b02700 -1 reset not still connected to 0x558c920cf520 2024-04-24T16:42:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 470 2024-04-24T16:42:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 472 2024-04-24T16:42:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 474 2024-04-24T16:42:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 476 2024-04-24T16:42:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 478 2024-04-24T16:42:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 480 2024-04-24T16:42:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:42:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:42:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:42:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:42:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:42:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:42:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:42:57.209 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T16:42:57.209+0000 7f7d32b02700 -1 reset not still connected to 0x558c920cf520 2024-04-24T16:42:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 482 2024-04-24T16:42:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 484 2024-04-24T16:43:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 486 2024-04-24T16:43:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 488 2024-04-24T16:43:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 490 2024-04-24T16:43:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 492 2024-04-24T16:43:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 494 2024-04-24T16:43:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 496 2024-04-24T16:43:32.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 498 2024-04-24T16:43:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 500 2024-04-24T16:43:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 502 2024-04-24T16:43:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 504 2024-04-24T16:43:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:43:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:43:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:43:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:43:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:43:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:43:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:43:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 506 2024-04-24T16:43:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:44:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:44:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:44:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:44:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:44:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:44:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:44:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 508 2024-04-24T16:44:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:44:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:44:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:44:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:44:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:44:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:44:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:44:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 510 2024-04-24T16:44:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:46:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T16:46:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T16:46:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 53424 2024-04-24T16:46:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T16:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:46:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:46:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=97376309d72253c3c5ef05acd5eaf9aa 2024-04-24T16:46:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T16:46:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T16:46:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:46:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:46:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:46:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T16:47:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T16:47:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T16:47:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:47:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:47:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:47:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:47:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:47:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:47:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:47:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:47:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:47:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:47:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:47:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:47:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:47:04.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:47:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:47:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:47:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:47:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:47:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:47:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:47:00 2024-04-24T16:47:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:47:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:47:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:47:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T16:47:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:47:01' 2024-04-24T16:47:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:47:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:47:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:47:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:47:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:47:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:47:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:47:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:47:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:47:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:47:00' 2024-04-24T16:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 16:47:00' 2024-04-24T16:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:47:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:47:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:47:00' 2024-04-24T16:47:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:47:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:47:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T16:47:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 16:47:00cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image demoted2024-04-24 16:47:01' 2024-04-24T16:47:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:47:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:47:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:47:00' 2024-04-24T16:47:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:47:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:47:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:47:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:47:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:47:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:47:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:47:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T16:47:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:47:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:47:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:47:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:47:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:47:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:47:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:47:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:47:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:47:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:47:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:47:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:47:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:47:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:47:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:47:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T16:47:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:47:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:47:01 2024-04-24T16:47:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:47:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:47:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:47:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:47:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:47:00' 2024-04-24T16:47:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:47:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:47:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:47:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image demoted 2024-04-24T16:47:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:47:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:47:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:47:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:47:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:47:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:47:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:47:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:47:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:47:01' 2024-04-24T16:47:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 16:47:01' 2024-04-24T16:47:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:47:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:47:01' 2024-04-24T16:47:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:47:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:47:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T16:47:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image demoted42614264mirror.0smithi1362024-04-24 16:47:01cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 16:47:00' 2024-04-24T16:47:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:47:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:47:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:47:01' 2024-04-24T16:47:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:47:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:47:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:47:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:47:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:47:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:47:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:47:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T16:47:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T16:47:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T16:47:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T16:47:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:47:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:47:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:47:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:47:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:47:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=97376309d72253c3c5ef05acd5eaf9aa 2024-04-24T16:47:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 97376309d72253c3c5ef05acd5eaf9aa != \9\7\3\7\6\3\0\9\d\7\2\2\5\3\c\3\c\5\e\f\0\5\a\c\d\5\e\a\f\9\a\a ]] 2024-04-24T16:47:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T16:47:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T16:47:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T16:47:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:47:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:47:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:47:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:47:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T16:47:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=54429 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:47:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:47:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:47:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 515 2024-04-24T16:47:29.745 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:47:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:47:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:47:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:47:34.743 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 517 2024-04-24T16:47:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:47:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:47:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:47:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:47:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 519 2024-04-24T16:47:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:47:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:47:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:47:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:47:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 521 2024-04-24T16:47:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:47:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:47:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:47:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:47:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 523 2024-04-24T16:47:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:47:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:47:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:47:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:47:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 525 2024-04-24T16:47:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:47:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:47:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:47:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:47:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:47:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:47:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:47:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 527 2024-04-24T16:47:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 529 2024-04-24T16:48:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 531 2024-04-24T16:48:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 533 2024-04-24T16:48:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 535 2024-04-24T16:48:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 537 2024-04-24T16:48:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 539 2024-04-24T16:48:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 541 2024-04-24T16:48:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 543 2024-04-24T16:48:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 545 2024-04-24T16:48:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 547 2024-04-24T16:48:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 549 2024-04-24T16:48:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:48:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:48:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:48:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:48:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:48:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:48:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:48:59.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 551 2024-04-24T16:48:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 553 2024-04-24T16:49:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:08.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 555 2024-04-24T16:49:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 557 2024-04-24T16:49:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 559 2024-04-24T16:49:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 561 2024-04-24T16:49:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 563 2024-04-24T16:49:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 565 2024-04-24T16:49:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:36.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 567 2024-04-24T16:49:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:43.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 569 2024-04-24T16:49:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 571 2024-04-24T16:49:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:49:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:49:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:49:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:49:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:49:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:49:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:49:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 573 2024-04-24T16:49:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:52:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T16:52:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T16:52:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 54429 2024-04-24T16:52:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T16:52:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:52:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:52:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=108aeb61fadc55dfe4b94af3e7611cce 2024-04-24T16:52:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T16:52:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T16:52:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:52:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:52:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:52:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T16:52:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T16:52:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T16:52:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:52:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:52:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:52:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:52:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:52:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:52:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:52:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:52:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:52:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:52:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:52:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:52:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:52:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:52:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:52:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:52:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:52:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+stopped 2024-04-24T16:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: local image is primary 2024-04-24T16:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:52:45 2024-04-24T16:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:52:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:52:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T16:52:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:52:49' 2024-04-24T16:52:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:52:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 1 2024-04-24T16:52:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:52:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 2 2024-04-24T16:52:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:52:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:52:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:52:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:52:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:52:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:52:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:52:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:52:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:52:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:52:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:52:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:52:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:52:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:52:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:52:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:52:50 2024-04-24T16:52:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:52:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:52:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:52:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:52:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:52:50' 2024-04-24T16:52:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:52:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:52:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:52:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:52:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:52:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:52:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:52:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:52:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:52:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:52:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:52:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:52:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:52:50' 2024-04-24T16:52:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 16:52:50' 2024-04-24T16:52:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:52:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:52:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:52:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:52:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:52:50' 2024-04-24T16:52:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:52:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:52:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T16:52:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 16:52:50cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 16:52:50' 2024-04-24T16:52:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:52:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:52:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:52:50' 2024-04-24T16:52:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:52:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:52:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:52:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:52:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:52:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:52:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:52:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T16:52:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:52:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:52:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:52:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:52:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:52:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:52:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:52:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:52:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:52:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:52:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:52:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:52:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:52:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:52:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:52:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:52:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:52:54.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:52:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:52:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:52:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:52:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:52:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:52:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:52:50 2024-04-24T16:52:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:52:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:52:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:52:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:52:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:52:50' 2024-04-24T16:52:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:52:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:52:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:52:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:52:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:52:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:52:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:52:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:52:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:52:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:52:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:52:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:52:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:52:50' 2024-04-24T16:52:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 16:52:50' 2024-04-24T16:52:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:52:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:52:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:52:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:52:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:52:50' 2024-04-24T16:52:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:52:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:52:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T16:52:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 16:52:50cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 16:52:50' 2024-04-24T16:52:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:52:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:52:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:52:50' 2024-04-24T16:52:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:52:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:52:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:52:54.183 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:52:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:52:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:52:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:52:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T16:52:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:52:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:52:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:52:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T16:52:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T16:52:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T16:52:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:52:54.279 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:52:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:52:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:52:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:52:55.451 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T16:52:55.447+0000 7f7d32b02700 -1 reset not still connected to 0x558ca4ba4270 2024-04-24T16:53:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=108aeb61fadc55dfe4b94af3e7611cce 2024-04-24T16:53:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 108aeb61fadc55dfe4b94af3e7611cce != \1\0\8\a\e\b\6\1\f\a\d\c\5\5\d\f\e\4\b\9\4\a\f\3\e\7\6\1\1\c\c\e ]] 2024-04-24T16:53:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T16:53:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T16:53:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T16:53:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:53:12.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:53:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:53:13.462 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:53:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T16:53:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=55437 2024-04-24T16:53:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:53:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:53:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T16:53:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:53:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:53:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:53:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:53:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:53:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:53:15.446 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 581 2024-04-24T16:53:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:53:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:53:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:53:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:53:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 583 2024-04-24T16:53:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:53:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:53:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:53:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:53:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 585 2024-04-24T16:53:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:53:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:53:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:53:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:53:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 587 2024-04-24T16:53:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:53:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:53:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:53:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:53:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 589 2024-04-24T16:53:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:53:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:53:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:53:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:53:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 591 2024-04-24T16:53:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:53:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:53:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:53:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:53:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 593 2024-04-24T16:53:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:53:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:53:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:53:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:53:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 595 2024-04-24T16:53:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:53:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:53:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:53:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:53:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 597 2024-04-24T16:53:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:53:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:53:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:53:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:53:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:53:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:53:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 599 2024-04-24T16:54:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 601 2024-04-24T16:54:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 603 2024-04-24T16:54:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 605 2024-04-24T16:54:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 607 2024-04-24T16:54:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 609 2024-04-24T16:54:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 611 2024-04-24T16:54:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 613 2024-04-24T16:54:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:40.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 615 2024-04-24T16:54:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 617 2024-04-24T16:54:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 619 2024-04-24T16:54:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:53.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:53.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:53.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:53.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:53.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:54:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 621 2024-04-24T16:54:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:54:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:54:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:54:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:54:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:54:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:54:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:55:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 623 2024-04-24T16:55:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:55:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:55:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 625 2024-04-24T16:55:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:55:08.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:55:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:55:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:55:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:55:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:55:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:55:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 627 2024-04-24T16:55:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:55:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:55:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:55:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:55:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:55:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:55:13.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:55:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 629 2024-04-24T16:55:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:55:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:55:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:55:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:55:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:55:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:55:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:55:20.293 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T16:55:20.292+0000 7f7d32b02700 -1 reset not still connected to 0x558ca4ba4270 2024-04-24T16:55:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 631 2024-04-24T16:55:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:55:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:55:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:55:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:55:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:55:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:55:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:55:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 633 2024-04-24T16:55:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:55:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:55:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:55:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:55:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:55:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:55:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:55:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 635 2024-04-24T16:55:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:55:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:55:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:55:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:55:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:55:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:55:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:55:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 637 2024-04-24T16:55:35.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:55:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:55:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T16:55:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:55:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:55:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:55:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:55:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 639 2024-04-24T16:55:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:58:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T16:58:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T16:58:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 55437 2024-04-24T16:58:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T16:58:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:58:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:58:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=d09d4a7ea16b8190d38eb63f548c1032 2024-04-24T16:58:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T16:58:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T16:58:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:58:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:58:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:58:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T16:58:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T16:58:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T16:58:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:58:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:58:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:58:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:58:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:58:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:58:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:58:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:58:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:58:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:58:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:58:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:58:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:58:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:58:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T16:58:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:58:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:58:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:58:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:58:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:58:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:58:35 2024-04-24T16:58:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:58:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T16:58:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:58:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:58:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:58:35' 2024-04-24T16:58:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:58:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:58:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:58:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:58:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:58:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:58:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:58:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:58:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:58:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:58:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:58:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:58:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:58:35' 2024-04-24T16:58:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 16:58:35' 2024-04-24T16:58:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T16:58:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:58:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:58:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:58:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:58:35' 2024-04-24T16:58:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:58:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:58:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T16:58:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 16:58:35cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 16:58:35' 2024-04-24T16:58:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:58:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:58:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:58:35' 2024-04-24T16:58:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:58:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:58:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:58:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:58:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T16:58:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T16:58:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T16:58:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:58:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:58:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:58:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:58:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T16:58:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T16:58:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:58:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:58:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T16:58:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T16:58:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T16:58:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:58:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:58:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T16:58:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:58:35 2024-04-24T16:58:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T16:58:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T16:58:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T16:58:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T16:58:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 16:58:35' 2024-04-24T16:58:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T16:58:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T16:58:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T16:58:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T16:58:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T16:58:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:58:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T16:58:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:58:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T16:58:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T16:58:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T16:58:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T16:58:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:58:35' 2024-04-24T16:58:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 16:58:35' 2024-04-24T16:58:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:58:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:58:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:58:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T16:58:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 16:58:35' 2024-04-24T16:58:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T16:58:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T16:58:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T16:58:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 16:58:35cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 16:58:35' 2024-04-24T16:58:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T16:58:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T16:58:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 16:58:35' 2024-04-24T16:58:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T16:58:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T16:58:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T16:58:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T16:58:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T16:58:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:58:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T16:58:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T16:58:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:58:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:58:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:58:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T16:58:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T16:58:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T16:58:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T16:58:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T16:58:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T16:58:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T16:58:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T16:58:41.251 INFO:tasks.ceph.cluster1.osd.2.smithi042.stderr:2024-04-24T16:58:41.249+0000 7f28a8be5700 -1 reset not still connected to 0x56326f3752b0 2024-04-24T16:58:41.251 INFO:tasks.ceph.cluster1.osd.2.smithi042.stderr:2024-04-24T16:58:41.249+0000 7f28a8be5700 -1 reset not still connected to 0x563275263790 2024-04-24T16:58:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=d09d4a7ea16b8190d38eb63f548c1032 2024-04-24T16:58:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ d09d4a7ea16b8190d38eb63f548c1032 != \d\0\9\d\4\a\7\e\a\1\6\b\8\1\9\0\d\3\8\e\b\6\3\f\5\4\8\c\1\0\3\2 ]] 2024-04-24T16:58:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T16:58:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T16:58:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T16:58:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T16:58:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T16:58:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T16:58:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T16:58:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T16:58:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=56446 2024-04-24T16:58:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T16:58:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T16:58:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T16:58:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T16:58:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:58:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:58:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:58:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:58:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:58:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:58:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:58:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:58:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:58:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T16:58:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T16:59:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 643 2024-04-24T16:59:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 645 2024-04-24T16:59:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 647 2024-04-24T16:59:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:15.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 649 2024-04-24T16:59:15.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 651 2024-04-24T16:59:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 653 2024-04-24T16:59:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 655 2024-04-24T16:59:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 657 2024-04-24T16:59:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 659 2024-04-24T16:59:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 661 2024-04-24T16:59:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 663 2024-04-24T16:59:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:55.745 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 665 2024-04-24T16:59:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T16:59:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T16:59:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T16:59:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T16:59:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T16:59:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T16:59:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T16:59:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 667 2024-04-24T16:59:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 669 2024-04-24T17:00:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 671 2024-04-24T17:00:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 673 2024-04-24T17:00:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 675 2024-04-24T17:00:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 677 2024-04-24T17:00:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 679 2024-04-24T17:00:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 681 2024-04-24T17:00:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 683 2024-04-24T17:00:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 685 2024-04-24T17:00:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 687 2024-04-24T17:00:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 689 2024-04-24T17:00:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:00:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:00:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:00:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:00:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:00:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:00:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:00:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 691 2024-04-24T17:00:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:01:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:01:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:01:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:01:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:01:02.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:01:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:01:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 693 2024-04-24T17:01:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:01:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:01:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:01:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:01:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:01:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:01:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:01:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 695 2024-04-24T17:01:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:01:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:01:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:01:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:01:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:01:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:01:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:01:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 697 2024-04-24T17:01:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:01:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:01:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 699 2024-04-24T17:01:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:01:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:01:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:01:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:01:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:01:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:01:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:01:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 701 2024-04-24T17:01:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:03:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T17:03:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T17:03:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 56446 2024-04-24T17:03:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T17:03:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:03:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:04:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=81a8b9299ffc13bd8a1ee4e36830ac75 2024-04-24T17:04:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T17:04:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T17:04:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:04:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T17:04:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T17:04:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T17:04:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:04:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:04:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:04:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:04:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:04:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:04:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:04:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:04:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:04:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:04:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:04:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:04:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:04:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:04:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:04:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:04:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:04:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:04:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:04:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:04:20 2024-04-24T17:04:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:04:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:04:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:04:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:04:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:04:20' 2024-04-24T17:04:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:04:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:04:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:04:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:04:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:04:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:04:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:04:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:04:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:04:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:04:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:04:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:04:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:04:20' 2024-04-24T17:04:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 17:04:20' 2024-04-24T17:04:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:04:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:04:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:04:20' 2024-04-24T17:04:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:04:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:04:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T17:04:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 17:04:20cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 17:04:20' 2024-04-24T17:04:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:04:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:04:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:04:20' 2024-04-24T17:04:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:04:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:04:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:04:22.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:04:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:04:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:04:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:04:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T17:04:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:04:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:04:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:04:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:04:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:04:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:04:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:04:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:04:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:04:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:04:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:04:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:04:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:04:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:04:20 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:04:20' 2024-04-24T17:04:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:04:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:04:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:04:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:04:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:04:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:04:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:04:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:04:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:04:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:04:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:04:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:04:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:04:20' 2024-04-24T17:04:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 17:04:20' 2024-04-24T17:04:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:04:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:04:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:04:20' 2024-04-24T17:04:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:04:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:04:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T17:04:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 17:04:20cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 17:04:20' 2024-04-24T17:04:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:04:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:04:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:04:20' 2024-04-24T17:04:23.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:04:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:04:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:04:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:04:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:04:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:04:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:04:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T17:04:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:04:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T17:04:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T17:04:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T17:04:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T17:04:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T17:04:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T17:04:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:04:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:04:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=81a8b9299ffc13bd8a1ee4e36830ac75 2024-04-24T17:04:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 81a8b9299ffc13bd8a1ee4e36830ac75 != \8\1\a\8\b\9\2\9\9\f\f\c\1\3\b\d\8\a\1\e\e\4\e\3\6\8\3\0\a\c\7\5 ]] 2024-04-24T17:04:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T17:04:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T17:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T17:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T17:04:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T17:04:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T17:04:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T17:04:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T17:04:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=57528 2024-04-24T17:04:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T17:04:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:04:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T17:04:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T17:04:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 709 2024-04-24T17:04:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:04:51.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:04:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:04:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:04:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:04:53.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 711 2024-04-24T17:04:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:04:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:04:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:04:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:04:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:04:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:04:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:04:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 713 2024-04-24T17:04:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 715 2024-04-24T17:05:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 717 2024-04-24T17:05:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:11.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 719 2024-04-24T17:05:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 721 2024-04-24T17:05:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 723 2024-04-24T17:05:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 725 2024-04-24T17:05:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:32.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 727 2024-04-24T17:05:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:35.802 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 729 2024-04-24T17:05:37.408 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 731 2024-04-24T17:05:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 733 2024-04-24T17:05:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 735 2024-04-24T17:05:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:05:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:05:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:05:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:05:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:05:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:05:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:05:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 737 2024-04-24T17:05:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 739 2024-04-24T17:06:02.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 741 2024-04-24T17:06:07.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 743 2024-04-24T17:06:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:17.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 745 2024-04-24T17:06:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 747 2024-04-24T17:06:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 749 2024-04-24T17:06:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 751 2024-04-24T17:06:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:37.406 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 753 2024-04-24T17:06:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:40.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 755 2024-04-24T17:06:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 757 2024-04-24T17:06:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 759 2024-04-24T17:06:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:06:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:06:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:06:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:06:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:06:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:06:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:06:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 761 2024-04-24T17:06:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:07:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:07:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:07:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:07:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:07:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:07:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:07:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 763 2024-04-24T17:07:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:07:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:07:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:07:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:07:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:07:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:07:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:07:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 765 2024-04-24T17:07:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:07:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:07:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:07:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:07:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:07:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:07:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:07:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 767 2024-04-24T17:07:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:09:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T17:09:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T17:09:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 57528 2024-04-24T17:09:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T17:09:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:09:46.932 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:10:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=5167be05584673c0c8a7a526cb5f98bf 2024-04-24T17:10:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T17:10:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T17:10:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:10:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T17:10:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T17:10:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T17:10:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:10:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:10:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:10:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:10:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:10:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:10:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:10:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:10:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:10:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:10:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:10:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:10:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:10:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:10:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:10:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:10:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+stopped 2024-04-24T17:10:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: local image is primary 2024-04-24T17:10:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:10:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:10:05 2024-04-24T17:10:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:10:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:10:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:10:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:10:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:10:08' 2024-04-24T17:10:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:10:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 1 2024-04-24T17:10:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:10:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 2 2024-04-24T17:10:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:10:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:10:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:10:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:10:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:10:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:10:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:10:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:10:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:10:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:10:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:10:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:10:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:10:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:10:10 2024-04-24T17:10:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:10:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:10:11.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:10:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:10:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:10:10' 2024-04-24T17:10:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:10:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:10:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:10:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:10:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:10:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:10:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:10:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:10:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:10:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:10:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:10:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:10:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:10:10' 2024-04-24T17:10:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 17:10:10' 2024-04-24T17:10:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:10:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:10:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:10:10' 2024-04-24T17:10:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:10:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:10:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T17:10:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 17:10:10cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 17:10:10' 2024-04-24T17:10:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:10:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:10:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:10:10' 2024-04-24T17:10:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:10:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:10:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:10:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:10:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:10:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:10:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:10:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T17:10:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:10:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:10:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:10:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:10:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:10:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:10:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:10:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:10:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:10:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:10:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:10:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:10:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:10:10 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:10:10' 2024-04-24T17:10:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:10:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:10:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:10:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:10:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:10:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:10:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:10:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:10:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:10:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:10:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:10:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:10:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:10:10' 2024-04-24T17:10:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 17:10:10' 2024-04-24T17:10:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:10:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:10:10' 2024-04-24T17:10:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:10:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:10:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T17:10:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 17:10:10cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 17:10:10' 2024-04-24T17:10:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:10:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:10:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:10:10' 2024-04-24T17:10:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:10:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:10:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:10:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:10:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:10:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:10:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:10:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T17:10:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T17:10:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T17:10:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T17:10:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T17:10:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T17:10:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T17:10:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:10:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:10:31.617 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T17:10:31.617+0000 7f7d32b02700 -1 reset not still connected to 0x558c888b9e10 2024-04-24T17:10:31.617 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T17:10:31.617+0000 7f7d32b02700 -1 reset not still connected to 0x558c920cf520 2024-04-24T17:10:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=5167be05584673c0c8a7a526cb5f98bf 2024-04-24T17:10:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 5167be05584673c0c8a7a526cb5f98bf != \5\1\6\7\b\e\0\5\5\8\4\6\7\3\c\0\c\8\a\7\a\5\2\6\c\b\5\f\9\8\b\f ]] 2024-04-24T17:10:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T17:10:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T17:10:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T17:10:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T17:10:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T17:10:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T17:10:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T17:10:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=58507 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:10:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T17:10:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T17:10:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 771 2024-04-24T17:10:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:10:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:10:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:10:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:10:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 773 2024-04-24T17:10:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:10:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:10:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:10:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:10:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 775 2024-04-24T17:10:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:10:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:10:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:10:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:10:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 777 2024-04-24T17:10:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:10:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:10:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:10:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:10:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 779 2024-04-24T17:10:56.870 INFO:tasks.ceph.cluster1.osd.1.smithi042.stderr:2024-04-24T17:10:56.870+0000 7f2d334fc700 -1 reset not still connected to 0x5648b9f351e0 2024-04-24T17:10:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:10:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:10:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:10:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:10:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:10:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:10:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 781 2024-04-24T17:11:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 783 2024-04-24T17:11:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 785 2024-04-24T17:11:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 787 2024-04-24T17:11:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 789 2024-04-24T17:11:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 791 2024-04-24T17:11:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 793 2024-04-24T17:11:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 795 2024-04-24T17:11:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 797 2024-04-24T17:11:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 799 2024-04-24T17:11:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 801 2024-04-24T17:11:51.927 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:11:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 803 2024-04-24T17:11:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:11:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:11:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:11:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:11:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:11:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:11:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 805 2024-04-24T17:12:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 807 2024-04-24T17:12:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 809 2024-04-24T17:12:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 811 2024-04-24T17:12:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 813 2024-04-24T17:12:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:26.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 815 2024-04-24T17:12:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 817 2024-04-24T17:12:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 819 2024-04-24T17:12:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 821 2024-04-24T17:12:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 823 2024-04-24T17:12:46.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 825 2024-04-24T17:12:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:12:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 827 2024-04-24T17:12:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:12:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:12:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:12:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:12:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:12:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:12:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:13:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 829 2024-04-24T17:13:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:15:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T17:15:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T17:15:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 58507 2024-04-24T17:15:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T17:15:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:15:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:15:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=d3ccb5d50d568206931435db6f090752 2024-04-24T17:15:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T17:15:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T17:15:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:15:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:15:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:15:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T17:15:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T17:15:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T17:15:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:15:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:15:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:15:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:15:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:15:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:15:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:15:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:15:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:15:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:15:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:15:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:15:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:15:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:15:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:15:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:15:58.049 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:15:58.049 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:15:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:15:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:15:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:15:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:15:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:15:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:15:56 2024-04-24T17:15:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:15:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:15:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:15:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:15:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:15:56' 2024-04-24T17:15:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:15:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:15:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:15:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:15:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:15:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:15:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:15:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:15:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:15:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:15:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:15:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:15:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:15:56' 2024-04-24T17:15:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 17:15:56' 2024-04-24T17:15:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:15:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:15:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:15:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:15:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:15:56' 2024-04-24T17:15:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:15:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:15:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T17:15:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 17:15:56cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image demoted2024-04-24 17:15:56' 2024-04-24T17:15:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:15:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:15:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:15:56' 2024-04-24T17:15:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:15:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:15:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:15:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:15:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:15:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:15:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:15:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:15:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:15:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:15:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:15:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:15:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:15:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:15:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:15:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:15:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:15:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:15:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:15:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:15:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:15:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:15:56 2024-04-24T17:15:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:15:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:15:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:15:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:15:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:15:56' 2024-04-24T17:15:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:15:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:15:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:15:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image demoted 2024-04-24T17:15:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:15:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:15:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:15:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:15:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:15:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:15:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:15:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:15:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:15:56' 2024-04-24T17:15:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 17:15:56' 2024-04-24T17:15:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:15:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:15:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:15:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:15:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:15:56' 2024-04-24T17:15:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:15:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:15:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T17:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image demoted42594265mirror.0smithi1362024-04-24 17:15:56cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 17:15:56' 2024-04-24T17:15:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:15:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:15:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:15:56' 2024-04-24T17:15:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:15:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:15:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:15:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:15:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:15:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:15:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:15:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T17:15:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:15:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:15:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:15:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T17:15:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T17:16:00.773 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T17:16:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T17:16:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T17:16:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T17:16:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:16:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:16:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=d3ccb5d50d568206931435db6f090752 2024-04-24T17:16:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ d3ccb5d50d568206931435db6f090752 != \d\3\c\c\b\5\d\5\0\d\5\6\8\2\0\6\9\3\1\4\3\5\d\b\6\f\0\9\0\7\5\2 ]] 2024-04-24T17:16:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T17:16:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T17:16:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T17:16:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T17:16:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T17:16:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T17:16:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T17:16:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T17:16:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=59460 2024-04-24T17:16:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T17:16:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T17:16:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T17:16:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T17:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:16:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:16:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T17:16:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T17:16:22.747 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 837 2024-04-24T17:16:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:16:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:16:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:16:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:16:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:16:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:16:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:16:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 839 2024-04-24T17:16:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:16:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:16:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:16:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:16:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:16:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:16:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:16:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 841 2024-04-24T17:16:32.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:16:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:16:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:16:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:16:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:16:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:16:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:16:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 843 2024-04-24T17:16:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:16:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:16:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:16:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:16:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:16:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:16:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:16:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 845 2024-04-24T17:16:42.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:16:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:16:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:16:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:16:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:16:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:16:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:16:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 847 2024-04-24T17:16:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:16:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:16:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:16:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:16:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:16:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:16:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:16:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 849 2024-04-24T17:16:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:16:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:16:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:16:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:16:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:16:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:16:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:16:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 851 2024-04-24T17:16:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 853 2024-04-24T17:17:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 855 2024-04-24T17:17:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 857 2024-04-24T17:17:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 859 2024-04-24T17:17:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:21.524 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T17:17:21.524+0000 7f7d32b02700 -1 reset not still connected to 0x558c920cf520 2024-04-24T17:17:22.523 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 861 2024-04-24T17:17:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 863 2024-04-24T17:17:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 865 2024-04-24T17:17:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 867 2024-04-24T17:17:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 869 2024-04-24T17:17:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 871 2024-04-24T17:17:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 873 2024-04-24T17:17:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:17:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:17:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:17:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:17:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:17:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:17:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:17:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 875 2024-04-24T17:17:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:18:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:18:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:18:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:18:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:18:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:18:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:18:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 877 2024-04-24T17:18:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:18:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:18:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:18:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:18:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:18:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:18:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:18:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 879 2024-04-24T17:18:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:18:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:18:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:18:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:18:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:18:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:18:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:18:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 881 2024-04-24T17:18:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:18:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:18:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:18:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:18:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:18:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:18:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:18:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 883 2024-04-24T17:18:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:18:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:18:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:18:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:18:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:18:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:18:20.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:18:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 885 2024-04-24T17:18:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:18:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:18:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:18:25.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:18:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:18:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:18:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:18:27.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 887 2024-04-24T17:18:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:18:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:18:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:18:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:18:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:18:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:18:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:18:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 889 2024-04-24T17:18:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:18:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:18:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:18:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:18:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:18:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:18:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:18:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 892 2024-04-24T17:18:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:18:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:18:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:18:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:18:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:18:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:18:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:18:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 894 2024-04-24T17:18:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:18:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:18:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:18:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:18:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:18:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:18:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:18:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 896 2024-04-24T17:18:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:21:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T17:21:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T17:21:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 59460 2024-04-24T17:21:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T17:21:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:21:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:21:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=92490f892ceab447b45f564c924df19e 2024-04-24T17:21:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T17:21:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T17:21:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:21:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:21:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:21:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T17:21:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T17:21:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T17:21:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:21:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:21:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:21:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:21:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:21:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:21:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:21:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:21:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:21:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:21:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:21:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:21:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:21:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:21:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:21:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:21:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:21:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:21:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:21:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:21:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:21:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:21:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:21:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:21:41 2024-04-24T17:21:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:21:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:21:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:21:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:21:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:21:42' 2024-04-24T17:21:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:21:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:21:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:21:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:21:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:21:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:21:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:21:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:21:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:21:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:21:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:21:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:21:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:21:41' 2024-04-24T17:21:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 17:21:41' 2024-04-24T17:21:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:21:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:21:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:21:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:21:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:21:41' 2024-04-24T17:21:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:21:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:21:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T17:21:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 17:21:41cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image demoted2024-04-24 17:21:42' 2024-04-24T17:21:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:21:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:21:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:21:41' 2024-04-24T17:21:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:21:45.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:21:45.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:21:45.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:21:45.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:21:45.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:21:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:21:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:21:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:21:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:21:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:21:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:21:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:21:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:21:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:21:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:21:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:21:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:21:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:21:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:21:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:21:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:21:46 2024-04-24T17:21:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:21:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:21:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:21:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:21:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:21:46' 2024-04-24T17:21:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:21:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:21:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:21:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:21:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:21:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:21:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:21:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:21:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:21:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:21:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:21:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:21:46.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:21:46' 2024-04-24T17:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 17:21:46' 2024-04-24T17:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:21:46' 2024-04-24T17:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:21:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T17:21:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 17:21:46cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 17:21:46' 2024-04-24T17:21:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:21:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:21:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:21:46' 2024-04-24T17:21:46.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:21:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:21:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:21:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:21:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:21:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:21:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:21:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T17:21:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:21:46.152 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:21:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:21:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T17:21:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T17:21:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T17:21:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T17:21:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T17:21:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T17:21:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:21:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:22:05.230 INFO:tasks.ceph.cluster1.osd.1.smithi042.stderr:2024-04-24T17:22:05.230+0000 7f2d334fc700 -1 reset not still connected to 0x5648b63e40d0 2024-04-24T17:22:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=92490f892ceab447b45f564c924df19e 2024-04-24T17:22:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 92490f892ceab447b45f564c924df19e != \9\2\4\9\0\f\8\9\2\c\e\a\b\4\4\7\b\4\5\f\5\6\4\c\9\2\4\d\f\1\9\e ]] 2024-04-24T17:22:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T17:22:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T17:22:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T17:22:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T17:22:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T17:22:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T17:22:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T17:22:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T17:22:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=60463 2024-04-24T17:22:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:22:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T17:22:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T17:22:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 899 2024-04-24T17:22:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:22:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:22:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 901 2024-04-24T17:22:15.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:22:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:22:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 903 2024-04-24T17:22:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:22:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:22:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 905 2024-04-24T17:22:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:22:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:22:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 907 2024-04-24T17:22:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:22:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:22:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 909 2024-04-24T17:22:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:22:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:22:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 911 2024-04-24T17:22:40.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:22:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:22:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 913 2024-04-24T17:22:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 915 2024-04-24T17:22:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:22:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:22:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 917 2024-04-24T17:22:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:22:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:22:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:22:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:22:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:22:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:22:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 919 2024-04-24T17:23:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:03.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 921 2024-04-24T17:23:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 923 2024-04-24T17:23:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 925 2024-04-24T17:23:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 927 2024-04-24T17:23:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 929 2024-04-24T17:23:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 931 2024-04-24T17:23:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 933 2024-04-24T17:23:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 935 2024-04-24T17:23:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 937 2024-04-24T17:23:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 939 2024-04-24T17:23:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:23:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 941 2024-04-24T17:23:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:23:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:23:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:23:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:23:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:23:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:23:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:24:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 943 2024-04-24T17:24:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:24:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:24:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:24:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:24:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:24:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:24:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:24:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 945 2024-04-24T17:24:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:24:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:24:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:24:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:24:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:24:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:24:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:24:10.209 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 947 2024-04-24T17:24:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:24:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:24:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:24:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:24:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:24:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:24:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:24:15.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 949 2024-04-24T17:24:15.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:24:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:24:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:24:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:24:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:24:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:24:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:24:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 951 2024-04-24T17:24:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:24:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:24:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:24:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:24:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:24:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:24:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:24:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 953 2024-04-24T17:24:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:24:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:24:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:24:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:24:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:24:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:24:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:24:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 955 2024-04-24T17:24:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:24:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:24:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:24:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:24:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:24:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:24:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:24:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 957 2024-04-24T17:24:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:25:11.595 INFO:tasks.ceph.cluster2.osd.2.smithi136.stderr:2024-04-24T17:25:11.595+0000 7f5637975700 -1 reset not still connected to 0x55eb996c1450 2024-04-24T17:25:11.595 INFO:tasks.ceph.cluster2.osd.2.smithi136.stderr:2024-04-24T17:25:11.595+0000 7f5637975700 -1 reset not still connected to 0x55eb996c1790 2024-04-24T17:27:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T17:27:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T17:27:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 60463 2024-04-24T17:27:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T17:27:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:27:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:27:11.012 INFO:tasks.ceph.cluster2.osd.2.smithi136.stderr:2024-04-24T17:27:11.012+0000 7f5637975700 -1 reset not still connected to 0x55eb996c1450 2024-04-24T17:27:11.012 INFO:tasks.ceph.cluster2.osd.2.smithi136.stderr:2024-04-24T17:27:11.012+0000 7f5637975700 -1 reset not still connected to 0x55eb996c1790 2024-04-24T17:27:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=78f98b6b9293b4aada0b7a03c62b232f 2024-04-24T17:27:26.590 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T17:27:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T17:27:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:27:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T17:27:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T17:27:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T17:27:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:27:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:27:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:27:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:27:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:27:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:27:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:27:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:27:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:27:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:27:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:27:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:27:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:27:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:27:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:27:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:27:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:27:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:27:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:27:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:27:31 2024-04-24T17:27:32.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:27:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:27:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:27:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:27:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:27:31' 2024-04-24T17:27:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:27:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:27:32.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:27:32.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:27:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:27:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:27:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:27:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:27:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:27:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:27:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:27:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:27:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:27:31' 2024-04-24T17:27:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 17:27:31' 2024-04-24T17:27:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:27:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:27:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:27:31' 2024-04-24T17:27:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:27:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:27:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T17:27:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 17:27:31cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 17:27:31' 2024-04-24T17:27:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:27:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:27:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:27:31' 2024-04-24T17:27:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:27:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:27:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:27:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:27:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:27:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:27:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:27:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T17:27:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:27:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:27:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:27:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:27:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:27:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:27:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:27:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:27:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:27:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:27:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:27:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:27:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:27:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:27:31 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:27:31' 2024-04-24T17:27:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:27:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:27:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:27:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:27:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:27:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:27:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:27:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:27:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:27:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:27:33.364 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:27:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:27:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:27:31' 2024-04-24T17:27:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 17:27:31' 2024-04-24T17:27:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:27:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:27:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:27:31' 2024-04-24T17:27:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:27:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:27:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T17:27:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 17:27:31cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 17:27:31' 2024-04-24T17:27:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:27:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:27:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:27:31' 2024-04-24T17:27:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:27:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:27:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:27:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:27:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:27:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:27:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:27:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T17:27:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:27:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T17:27:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T17:27:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T17:27:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T17:27:33.934 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T17:27:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T17:27:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:27:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:27:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=78f98b6b9293b4aada0b7a03c62b232f 2024-04-24T17:27:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 78f98b6b9293b4aada0b7a03c62b232f != \7\8\f\9\8\b\6\b\9\2\9\3\b\4\a\a\d\a\0\b\7\a\0\3\c\6\2\b\2\3\2\f ]] 2024-04-24T17:27:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T17:27:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T17:27:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T17:27:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T17:27:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T17:27:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T17:27:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T17:27:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T17:27:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=61475 2024-04-24T17:27:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T17:27:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T17:27:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T17:27:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T17:27:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:27:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:27:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:27:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:27:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:27:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T17:27:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T17:27:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 966 2024-04-24T17:27:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:27:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:27:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:27:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:27:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:27:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:27:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:27:59.940 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 968 2024-04-24T17:27:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 970 2024-04-24T17:28:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:05.884 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T17:28:05.884+0000 7f7d32b02700 -1 reset not still connected to 0x558c9cc74c30 2024-04-24T17:28:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 972 2024-04-24T17:28:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:11.018 INFO:tasks.ceph.cluster1.osd.1.smithi042.stderr:2024-04-24T17:28:11.018+0000 7f2d334fc700 -1 reset not still connected to 0x5648b8a7f930 2024-04-24T17:28:11.019 INFO:tasks.ceph.cluster1.osd.1.smithi042.stderr:2024-04-24T17:28:11.020+0000 7f2d334fc700 -1 reset not still connected to 0x5648d1e7fa00 2024-04-24T17:28:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 974 2024-04-24T17:28:13.942 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:17.599 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T17:28:17.599+0000 7f7d32b02700 -1 reset not still connected to 0x558c920cf520 2024-04-24T17:28:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 976 2024-04-24T17:28:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 978 2024-04-24T17:28:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 980 2024-04-24T17:28:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:30.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 982 2024-04-24T17:28:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 984 2024-04-24T17:28:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 986 2024-04-24T17:28:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 988 2024-04-24T17:28:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 990 2024-04-24T17:28:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:28:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 992 2024-04-24T17:28:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:28:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:28:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:28:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:28:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:28:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:28:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 994 2024-04-24T17:29:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 996 2024-04-24T17:29:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 998 2024-04-24T17:29:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1000 2024-04-24T17:29:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1002 2024-04-24T17:29:21.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1004 2024-04-24T17:29:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1006 2024-04-24T17:29:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1008 2024-04-24T17:29:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1010 2024-04-24T17:29:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1012 2024-04-24T17:29:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:50.633 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T17:29:50.633+0000 7f7d32b02700 -1 reset not still connected to 0x558c920cf520 2024-04-24T17:29:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1014 2024-04-24T17:29:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:29:55.627 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T17:29:55.626+0000 7f7d32b02700 -1 reset not still connected to 0x558c920cf520 2024-04-24T17:29:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1016 2024-04-24T17:29:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:29:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:29:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:29:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:29:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:29:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:29:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:30:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1018 2024-04-24T17:30:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:30:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:30:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:30:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:30:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:30:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:30:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:30:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1020 2024-04-24T17:30:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:30:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:30:09.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:30:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:30:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:30:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:30:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:30:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1022 2024-04-24T17:30:11.644 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:30:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:30:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:30:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:30:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:30:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:30:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:30:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1024 2024-04-24T17:30:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T17:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T17:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 61475 2024-04-24T17:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T17:32:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:32:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:33:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=216352c40358220b3bed11fb9c70be7b 2024-04-24T17:33:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T17:33:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T17:33:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:33:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T17:33:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T17:33:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T17:33:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:33:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:33:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:33:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:33:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:33:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:33:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:33:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:33:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:33:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:33:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:33:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:33:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:33:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:33:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:33:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:33:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:33:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:33:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:33:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:33:16 2024-04-24T17:33:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:33:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:33:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:33:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:33:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:33:16' 2024-04-24T17:33:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:33:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:33:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:33:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:33:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:33:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:33:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:33:17.308 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:33:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:33:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:33:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:33:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:33:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:33:16' 2024-04-24T17:33:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 17:33:16' 2024-04-24T17:33:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:33:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:33:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:33:16' 2024-04-24T17:33:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:33:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:33:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T17:33:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 17:33:16cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image demoted2024-04-24 17:33:16' 2024-04-24T17:33:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:33:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:33:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:33:16' 2024-04-24T17:33:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:33:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:33:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:33:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:33:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:33:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:33:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:33:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T17:33:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:33:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:33:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:33:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:33:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:33:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:33:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:33:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:33:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:33:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:33:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:33:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:33:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:33:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:33:16 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:33:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:33:16' 2024-04-24T17:33:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:33:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:33:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:33:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image demoted 2024-04-24T17:33:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:33:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:33:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:33:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:33:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:33:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:33:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:33:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:33:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:33:16' 2024-04-24T17:33:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 17:33:16' 2024-04-24T17:33:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:33:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:33:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:33:16' 2024-04-24T17:33:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:33:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:33:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T17:33:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image demoted42614264mirror.0smithi1362024-04-24 17:33:16cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 17:33:16' 2024-04-24T17:33:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:33:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:33:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:33:16' 2024-04-24T17:33:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:33:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:33:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:33:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:33:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:33:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:33:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:33:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T17:33:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:33:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T17:33:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T17:33:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T17:33:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T17:33:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T17:33:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T17:33:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:33:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:33:21.590 INFO:tasks.ceph.cluster1.osd.1.smithi042.stderr:2024-04-24T17:33:21.590+0000 7f2d334fc700 -1 reset not still connected to 0x5648d1ede1a0 2024-04-24T17:33:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=216352c40358220b3bed11fb9c70be7b 2024-04-24T17:33:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 216352c40358220b3bed11fb9c70be7b != \2\1\6\3\5\2\c\4\0\3\5\8\2\2\0\b\3\b\e\d\1\1\f\b\9\c\7\0\b\e\7\b ]] 2024-04-24T17:33:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T17:33:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T17:33:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T17:33:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T17:33:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T17:33:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T17:33:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T17:33:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T17:33:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=62466 2024-04-24T17:33:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T17:33:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T17:33:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T17:33:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T17:33:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:33:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:33:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T17:33:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T17:33:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1027 2024-04-24T17:33:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:33:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:33:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:33:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:33:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:33:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1029 2024-04-24T17:33:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:33:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:33:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:33:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:33:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:33:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1031 2024-04-24T17:33:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:33:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:33:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:33:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:33:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:33:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1033 2024-04-24T17:33:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:33:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:33:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:33:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:33:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:33:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:33:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1035 2024-04-24T17:34:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1037 2024-04-24T17:34:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1039 2024-04-24T17:34:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1041 2024-04-24T17:34:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:18.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1043 2024-04-24T17:34:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1045 2024-04-24T17:34:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1047 2024-04-24T17:34:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:33.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1049 2024-04-24T17:34:35.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:38.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1051 2024-04-24T17:34:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1053 2024-04-24T17:34:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1055 2024-04-24T17:34:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:34:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1057 2024-04-24T17:34:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:34:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:34:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:34:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:34:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:34:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:34:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1059 2024-04-24T17:35:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1061 2024-04-24T17:35:05.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1063 2024-04-24T17:35:10.364 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1065 2024-04-24T17:35:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:18.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1067 2024-04-24T17:35:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1069 2024-04-24T17:35:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1071 2024-04-24T17:35:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:33.399 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1073 2024-04-24T17:35:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:40.408 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1075 2024-04-24T17:35:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:43.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1077 2024-04-24T17:35:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1079 2024-04-24T17:35:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:35:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1081 2024-04-24T17:35:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:35:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:35:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:35:58.453 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:35:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:35:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:35:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:36:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1083 2024-04-24T17:36:00.470 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:36:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:36:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:36:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:36:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:36:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:36:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:36:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1085 2024-04-24T17:36:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T17:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T17:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 62466 2024-04-24T17:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T17:38:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:38:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:38:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=bdc68e57f171e1bc75925d6a805bdf9e 2024-04-24T17:38:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T17:38:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T17:38:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:38:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:38:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:38:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T17:39:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T17:39:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T17:39:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:39:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:39:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:39:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:39:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:39:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:39:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:39:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:39:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:39:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:39:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:39:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:39:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:39:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:39:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:39:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:39:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:39:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:39:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:39:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:39:01 2024-04-24T17:39:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:39:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:39:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:39:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:39:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:39:01' 2024-04-24T17:39:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:39:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:39:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:39:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:39:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:39:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:39:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:39:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:39:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:39:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:39:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:39:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:39:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:39:01' 2024-04-24T17:39:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 17:39:01' 2024-04-24T17:39:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:39:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:39:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:39:01' 2024-04-24T17:39:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:39:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:39:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T17:39:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 17:39:01cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 17:39:01' 2024-04-24T17:39:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:39:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:39:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:39:01' 2024-04-24T17:39:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:39:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:39:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:39:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:39:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:39:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:39:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:39:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T17:39:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:39:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:39:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:39:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:39:02.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:39:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:39:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:39:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:39:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:39:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:39:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:39:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:39:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:39:01 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:39:01' 2024-04-24T17:39:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:39:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:39:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:39:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:39:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:39:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:39:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:39:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:39:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:39:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:39:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:39:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:39:03.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:39:01' 2024-04-24T17:39:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 17:39:01' 2024-04-24T17:39:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:39:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:39:01' 2024-04-24T17:39:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:39:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:39:03.562 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T17:39:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 17:39:01cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 17:39:01' 2024-04-24T17:39:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:39:03.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:39:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:39:01' 2024-04-24T17:39:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:39:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:39:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:39:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:39:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:39:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:39:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:39:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T17:39:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T17:39:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T17:39:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T17:39:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T17:39:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T17:39:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T17:39:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:39:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:39:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=bdc68e57f171e1bc75925d6a805bdf9e 2024-04-24T17:39:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ bdc68e57f171e1bc75925d6a805bdf9e != \b\d\c\6\8\e\5\7\f\1\7\1\e\1\b\c\7\5\9\2\5\d\6\a\8\0\5\b\d\f\9\e ]] 2024-04-24T17:39:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T17:39:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T17:39:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T17:39:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T17:39:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T17:39:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T17:39:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T17:39:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=63439 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:39:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:39:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T17:39:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T17:39:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1095 2024-04-24T17:39:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:39:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:39:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:39:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:29.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:39:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1097 2024-04-24T17:39:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:39:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:39:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:39:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:39:36.776 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1099 2024-04-24T17:39:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:39:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:39:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:39:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:39:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1101 2024-04-24T17:39:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:39:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:39:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:39:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:39:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1103 2024-04-24T17:39:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:39:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:39:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:39:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:39:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1105 2024-04-24T17:39:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:39:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:39:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:39:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:39:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1107 2024-04-24T17:39:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:39:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1109 2024-04-24T17:40:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1111 2024-04-24T17:40:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1113 2024-04-24T17:40:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:14.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1115 2024-04-24T17:40:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1117 2024-04-24T17:40:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:26.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1119 2024-04-24T17:40:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1121 2024-04-24T17:40:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1123 2024-04-24T17:40:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1125 2024-04-24T17:40:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1127 2024-04-24T17:40:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1129 2024-04-24T17:40:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:40:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1131 2024-04-24T17:40:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:40:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:40:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:40:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:40:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:40:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:40:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1133 2024-04-24T17:41:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:41:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:41:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:41:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:41:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:41:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:41:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1135 2024-04-24T17:41:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:41:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:41:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:41:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:41:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:41:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:41:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1137 2024-04-24T17:41:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:41:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1139 2024-04-24T17:41:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:41:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:41:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:41:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:41:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:41:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:41:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1141 2024-04-24T17:41:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:41:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:41:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:41:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:41:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:41:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:41:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1143 2024-04-24T17:41:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:41:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1145 2024-04-24T17:41:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:41:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:41:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:41:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:41:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:41:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:41:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1147 2024-04-24T17:41:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:41:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:41:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:41:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:41:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:41:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:41:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1149 2024-04-24T17:41:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:41:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:41:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:41:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:41:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:41:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:41:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1151 2024-04-24T17:41:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:41:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:41:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:41:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:41:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:41:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:41:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:41:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1153 2024-04-24T17:41:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:44:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T17:44:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T17:44:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 63439 2024-04-24T17:44:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T17:44:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:44:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:44:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=a6e049e26a895aa2d2960c30a5a45b75 2024-04-24T17:44:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T17:44:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T17:44:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:44:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:44:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:44:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T17:44:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T17:44:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T17:44:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:44:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:44:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:44:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:44:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:44:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:44:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:44:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:44:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:44:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:44:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:44:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:44:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:44:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:44:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:44:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:44:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:44:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:44:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:44:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:44:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:44:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:44:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:44:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:44:46 2024-04-24T17:44:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:44:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:44:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:44:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:44:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:44:47' 2024-04-24T17:44:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:44:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:44:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:44:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:44:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:44:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:44:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:44:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:44:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:44:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:44:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:44:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:44:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:44:46' 2024-04-24T17:44:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 17:44:46' 2024-04-24T17:44:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:44:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:44:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:44:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:44:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:44:46' 2024-04-24T17:44:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:44:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:44:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T17:44:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 17:44:46cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image demoted2024-04-24 17:44:47' 2024-04-24T17:44:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:44:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:44:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:44:46' 2024-04-24T17:44:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:44:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:44:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:44:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:44:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:44:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:44:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:44:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:44:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:44:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:44:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:44:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:44:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:44:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:44:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:44:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:44:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:44:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:44:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:44:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:44:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:44:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:44:47 2024-04-24T17:44:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:44:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:44:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:44:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:44:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:44:46' 2024-04-24T17:44:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:44:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:44:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:44:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image demoted 2024-04-24T17:44:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:44:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:44:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:44:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:44:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:44:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:44:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:44:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:44:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:44:47' 2024-04-24T17:44:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 17:44:47' 2024-04-24T17:44:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:44:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:44:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:44:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:44:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:44:47' 2024-04-24T17:44:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:44:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:44:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T17:44:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image demoted42614264mirror.0smithi1362024-04-24 17:44:47cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 17:44:46' 2024-04-24T17:44:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:44:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:44:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:44:47' 2024-04-24T17:44:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:44:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:44:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:44:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:44:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:44:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:44:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:44:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T17:44:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:44:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:44:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:44:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T17:44:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T17:44:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T17:44:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T17:44:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T17:44:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T17:44:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:44:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:45:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=a6e049e26a895aa2d2960c30a5a45b75 2024-04-24T17:45:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ a6e049e26a895aa2d2960c30a5a45b75 != \a\6\e\0\4\9\e\2\6\a\8\9\5\a\a\2\d\2\9\6\0\c\3\0\a\5\a\4\5\b\7\5 ]] 2024-04-24T17:45:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T17:45:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T17:45:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T17:45:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T17:45:09.954 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T17:45:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T17:45:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T17:45:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T17:45:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=64437 2024-04-24T17:45:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T17:45:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T17:45:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T17:45:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T17:45:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:45:11.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:45:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:45:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T17:45:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T17:45:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1155 2024-04-24T17:45:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:45:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:45:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:45:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:16.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:45:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1157 2024-04-24T17:45:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:45:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:45:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:45:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:45:23.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1159 2024-04-24T17:45:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:45:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:45:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:45:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:26.405 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:45:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1161 2024-04-24T17:45:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:45:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:45:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:45:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:45:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1163 2024-04-24T17:45:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:45:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:45:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:45:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:45:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1165 2024-04-24T17:45:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:45:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:45:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:45:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:45:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1167 2024-04-24T17:45:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:45:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:45:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:45:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:45:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1169 2024-04-24T17:45:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:45:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:45:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:45:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:45:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1172 2024-04-24T17:45:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:45:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:45:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:45:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1174 2024-04-24T17:45:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1176 2024-04-24T17:46:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1178 2024-04-24T17:46:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1180 2024-04-24T17:46:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:18.228 INFO:tasks.ceph.cluster1.osd.1.smithi042.stderr:2024-04-24T17:46:18.228+0000 7f2d334fc700 -1 reset not still connected to 0x5648cc8cd860 2024-04-24T17:46:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1182 2024-04-24T17:46:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1184 2024-04-24T17:46:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1186 2024-04-24T17:46:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1188 2024-04-24T17:46:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1190 2024-04-24T17:46:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1192 2024-04-24T17:46:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1194 2024-04-24T17:46:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1196 2024-04-24T17:46:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:46:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:46:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:46:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:46:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:46:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:46:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:46:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1198 2024-04-24T17:46:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:47:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:47:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:47:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:47:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:47:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:47:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:47:03.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1200 2024-04-24T17:47:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:47:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:47:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:47:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:47:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:47:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:47:06.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:47:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1202 2024-04-24T17:47:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:47:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:47:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:47:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:47:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:47:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:47:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:47:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1204 2024-04-24T17:47:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:47:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:47:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:47:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:47:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:47:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:47:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:47:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1206 2024-04-24T17:47:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:47:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:47:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:47:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:47:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:47:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:47:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:47:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1208 2024-04-24T17:47:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:47:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:47:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:47:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:47:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:47:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:47:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:47:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1210 2024-04-24T17:47:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:47:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:47:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:47:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:47:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:47:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:47:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:47:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1212 2024-04-24T17:47:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:47:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:47:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:47:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:47:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:47:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:47:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:47:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1214 2024-04-24T17:47:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:50:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T17:50:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T17:50:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 64437 2024-04-24T17:50:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T17:50:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:50:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:50:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=18caedfe60ca772138408561aff9f0f6 2024-04-24T17:50:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T17:50:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T17:50:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:50:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T17:50:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T17:50:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T17:50:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:50:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:50:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:50:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:50:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:50:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:50:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:50:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:50:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:50:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:50:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:50:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:50:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:50:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:50:34.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+stopped 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: local image is primary 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:50:31 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:50:33' 2024-04-24T17:50:34.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:50:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 1 2024-04-24T17:50:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:50:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 2 2024-04-24T17:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:50:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:50:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:50:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:50:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:50:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:50:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:50:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:50:36 2024-04-24T17:50:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:50:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:50:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:50:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:50:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:50:36' 2024-04-24T17:50:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:50:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:50:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:50:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:50:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:50:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:50:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:50:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:50:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:50:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:50:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:50:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:50:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:50:36' 2024-04-24T17:50:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 17:50:36' 2024-04-24T17:50:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:50:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:50:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:50:36' 2024-04-24T17:50:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:50:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:50:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T17:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 17:50:36cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 17:50:36' 2024-04-24T17:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:50:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:50:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:50:36' 2024-04-24T17:50:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T17:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:50:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:50:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:50:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:50:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:50:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:50:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:50:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:50:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:50:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:50:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:50:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:50:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:50:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:50:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:50:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:50:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:50:36 2024-04-24T17:50:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:50:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:50:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:50:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:50:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:50:36' 2024-04-24T17:50:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:50:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:50:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:50:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:50:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:50:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:50:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:50:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:50:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:50:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:50:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:50:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:50:36' 2024-04-24T17:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 17:50:36' 2024-04-24T17:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:50:36' 2024-04-24T17:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:50:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T17:50:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 17:50:36cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 17:50:36' 2024-04-24T17:50:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:50:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:50:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:50:36' 2024-04-24T17:50:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:50:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:50:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:50:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:50:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:50:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:50:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:50:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T17:50:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:50:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T17:50:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T17:50:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T17:50:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T17:50:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T17:50:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T17:50:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:50:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:50:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=18caedfe60ca772138408561aff9f0f6 2024-04-24T17:50:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 18caedfe60ca772138408561aff9f0f6 != \1\8\c\a\e\d\f\e\6\0\c\a\7\7\2\1\3\8\4\0\8\5\6\1\a\f\f\9\f\0\f\6 ]] 2024-04-24T17:50:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T17:50:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T17:50:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T17:50:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T17:50:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T17:50:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T17:50:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T17:50:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=65488 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:50:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:50:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:50:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:50:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:50:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T17:50:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T17:51:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1223 2024-04-24T17:51:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1225 2024-04-24T17:51:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1227 2024-04-24T17:51:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1229 2024-04-24T17:51:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1231 2024-04-24T17:51:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1233 2024-04-24T17:51:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1235 2024-04-24T17:51:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1237 2024-04-24T17:51:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1239 2024-04-24T17:51:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1241 2024-04-24T17:51:45.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1243 2024-04-24T17:51:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:51:55.865 INFO:tasks.ceph.cluster2.osd.1.smithi136.stderr:2024-04-24T17:51:55.870+0000 7f7d32b02700 -1 reset not still connected to 0x558ca1a771e0 2024-04-24T17:51:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1245 2024-04-24T17:51:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:51:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:51:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:51:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:51:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:51:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:51:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1247 2024-04-24T17:52:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1249 2024-04-24T17:52:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1251 2024-04-24T17:52:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1253 2024-04-24T17:52:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1255 2024-04-24T17:52:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1257 2024-04-24T17:52:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1259 2024-04-24T17:52:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1261 2024-04-24T17:52:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1263 2024-04-24T17:52:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1265 2024-04-24T17:52:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1267 2024-04-24T17:52:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:52:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1269 2024-04-24T17:52:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:52:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:52:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:52:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:52:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:52:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:52:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:53:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1271 2024-04-24T17:53:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:53:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:53:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:53:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:53:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:53:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:53:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:53:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1273 2024-04-24T17:53:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:53:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:53:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:53:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:53:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:53:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:53:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:53:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1275 2024-04-24T17:53:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:53:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:53:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:53:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:53:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:53:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:53:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:53:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1277 2024-04-24T17:53:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:53:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:53:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:53:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:53:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:53:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:53:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:53:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1279 2024-04-24T17:53:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:53:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:53:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T17:53:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:53:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:53:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:53:23.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:53:25.898 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1281 2024-04-24T17:53:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:55:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T17:55:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T17:55:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 65488 2024-04-24T17:55:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T17:56:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:56:00.070 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:56:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=640003ef3bcd103c55f02fe90eb85bb9 2024-04-24T17:56:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T17:56:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T17:56:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:56:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T17:56:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T17:56:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T17:56:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:56:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:56:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:56:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:56:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:56:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:56:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:56:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:56:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:56:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:56:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:56:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:56:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:56:23.044 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T17:56:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:56:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:56:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:56:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:56:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:56:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:56:21 2024-04-24T17:56:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:56:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T17:56:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:56:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:56:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:56:21' 2024-04-24T17:56:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:56:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:56:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:56:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T17:56:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:56:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:56:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:56:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:56:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:56:23.099 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:56:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:56:23.100 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:56:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:56:21' 2024-04-24T17:56:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 17:56:21' 2024-04-24T17:56:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T17:56:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:56:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:56:21' 2024-04-24T17:56:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:56:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:56:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T17:56:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 17:56:21cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image demoted2024-04-24 17:56:21' 2024-04-24T17:56:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:56:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:56:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:56:21' 2024-04-24T17:56:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:56:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:56:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:56:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:56:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:56:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:56:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:56:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T17:56:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:56:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:56:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:56:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:56:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T17:56:23.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T17:56:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T17:56:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:56:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:56:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T17:56:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T17:56:24.160 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:56:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:56:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T17:56:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T17:56:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T17:56:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:56:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T17:56:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T17:56:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:56:21 2024-04-24T17:56:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T17:56:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T17:56:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T17:56:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T17:56:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 17:56:21' 2024-04-24T17:56:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T17:56:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T17:56:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T17:56:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image demoted 2024-04-24T17:56:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T17:56:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:56:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T17:56:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:56:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T17:56:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T17:56:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T17:56:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T17:56:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:56:21' 2024-04-24T17:56:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 17:56:21' 2024-04-24T17:56:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:56:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T17:56:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 17:56:21' 2024-04-24T17:56:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T17:56:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T17:56:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T17:56:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image demoted42614264mirror.0smithi1362024-04-24 17:56:21cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 17:56:21' 2024-04-24T17:56:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T17:56:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T17:56:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 17:56:21' 2024-04-24T17:56:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T17:56:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T17:56:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T17:56:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T17:56:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T17:56:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:56:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T17:56:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T17:56:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:56:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T17:56:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T17:56:25.545 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T17:56:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T17:56:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T17:56:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T17:56:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T17:56:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T17:56:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=640003ef3bcd103c55f02fe90eb85bb9 2024-04-24T17:56:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 640003ef3bcd103c55f02fe90eb85bb9 != \6\4\0\0\0\3\e\f\3\b\c\d\1\0\3\c\5\5\f\0\2\f\e\9\0\e\b\8\5\b\b\9 ]] 2024-04-24T17:56:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T17:56:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T17:56:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T17:56:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T17:56:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T17:56:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T17:56:44.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T17:56:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T17:56:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=66466 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:45.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:56:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T17:56:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T17:56:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1284 2024-04-24T17:56:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:56:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:56:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:56:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:56:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:56:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1286 2024-04-24T17:56:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:56:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:56:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:56:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:56:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:56:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:56:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:56:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1288 2024-04-24T17:56:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1290 2024-04-24T17:57:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1292 2024-04-24T17:57:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1294 2024-04-24T17:57:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:16.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1296 2024-04-24T17:57:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1298 2024-04-24T17:57:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1300 2024-04-24T17:57:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:31.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1302 2024-04-24T17:57:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1304 2024-04-24T17:57:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1306 2024-04-24T17:57:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1308 2024-04-24T17:57:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1310 2024-04-24T17:57:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:57:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:57:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:57:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:57:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:57:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:57:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:57:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1312 2024-04-24T17:57:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:01.634 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1314 2024-04-24T17:58:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1316 2024-04-24T17:58:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:11.643 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1318 2024-04-24T17:58:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1320 2024-04-24T17:58:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1322 2024-04-24T17:58:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1324 2024-04-24T17:58:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1326 2024-04-24T17:58:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:37.385 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1328 2024-04-24T17:58:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1330 2024-04-24T17:58:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1332 2024-04-24T17:58:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1334 2024-04-24T17:58:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:58:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:58:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:58:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:58:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:58:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:58:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:58:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1336 2024-04-24T17:58:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:59:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:59:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:59:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:59:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:59:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:59:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:59:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1338 2024-04-24T17:59:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:59:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:59:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:59:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:59:08.578 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1340 2024-04-24T17:59:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T17:59:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T17:59:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T17:59:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T17:59:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T17:59:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T17:59:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T17:59:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1342 2024-04-24T17:59:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:01:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T18:01:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T18:01:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 66466 2024-04-24T18:01:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T18:01:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T18:01:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T18:02:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=81f1d6af8f2c450f7f7ac2eeb8e5b912 2024-04-24T18:02:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T18:02:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T18:02:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:02:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T18:02:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T18:02:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T18:02:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:02:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:02:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:02:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:02:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:02:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:02:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:02:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:02:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:02:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:02:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:02:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:02:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:02:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T18:02:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:02:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:02:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:02:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:02:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:02:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:02:06 2024-04-24T18:02:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:02:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T18:02:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:02:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T18:02:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:02:07' 2024-04-24T18:02:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:02:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T18:02:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T18:02:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T18:02:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T18:02:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:02:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T18:02:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:02:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T18:02:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T18:02:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T18:02:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T18:02:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:02:06' 2024-04-24T18:02:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 18:02:06' 2024-04-24T18:02:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:02:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:02:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 18:02:06' 2024-04-24T18:02:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:02:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:02:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T18:02:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 18:02:06cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image demoted2024-04-24 18:02:07' 2024-04-24T18:02:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T18:02:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T18:02:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:02:06' 2024-04-24T18:02:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T18:02:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T18:02:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:02:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T18:02:09.766 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:02:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:02:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:02:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T18:02:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:02:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:02:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:02:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:02:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:02:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:02:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:02:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:02:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:02:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:02:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:02:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T18:02:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:02:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:02:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:02:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T18:02:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:02:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:02:07 2024-04-24T18:02:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:02:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T18:02:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:02:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:02:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:02:06' 2024-04-24T18:02:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:02:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T18:02:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T18:02:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image demoted 2024-04-24T18:02:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T18:02:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:02:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T18:02:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:02:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T18:02:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T18:02:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T18:02:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T18:02:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:02:07' 2024-04-24T18:02:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 18:02:07' 2024-04-24T18:02:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:02:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 18:02:07' 2024-04-24T18:02:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:02:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:02:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T18:02:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image demoted42594265mirror.0smithi1362024-04-24 18:02:07cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 18:02:06' 2024-04-24T18:02:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T18:02:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T18:02:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:02:07' 2024-04-24T18:02:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T18:02:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T18:02:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:02:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T18:02:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:02:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:02:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:02:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T18:02:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T18:02:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T18:02:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T18:02:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T18:02:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T18:02:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T18:02:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T18:02:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T18:02:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=81f1d6af8f2c450f7f7ac2eeb8e5b912 2024-04-24T18:02:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 81f1d6af8f2c450f7f7ac2eeb8e5b912 != \8\1\f\1\d\6\a\f\8\f\2\c\4\5\0\f\7\f\7\a\c\2\e\e\b\8\e\5\b\9\1\2 ]] 2024-04-24T18:02:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T18:02:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T18:02:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T18:02:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T18:02:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T18:02:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T18:02:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T18:02:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T18:02:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=67527 2024-04-24T18:02:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T18:02:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T18:02:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T18:02:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T18:02:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:02:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:02:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:02:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T18:02:32.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T18:02:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1351 2024-04-24T18:02:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:02:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1353 2024-04-24T18:02:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:02:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:02:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:02:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:02:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1355 2024-04-24T18:02:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:02:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:02:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:02:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:02:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1357 2024-04-24T18:02:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:02:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:02:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:02:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:02:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1359 2024-04-24T18:02:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:02:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:02:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:02:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:02:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:02:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:02:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:02:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1361 2024-04-24T18:02:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1363 2024-04-24T18:03:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:07.038 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1365 2024-04-24T18:03:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1367 2024-04-24T18:03:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1369 2024-04-24T18:03:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1371 2024-04-24T18:03:24.096 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:29.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1373 2024-04-24T18:03:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1375 2024-04-24T18:03:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1377 2024-04-24T18:03:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1379 2024-04-24T18:03:44.128 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1381 2024-04-24T18:03:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1383 2024-04-24T18:03:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:03:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:03:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:03:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:03:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:03:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:03:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:03:59.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1385 2024-04-24T18:03:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1387 2024-04-24T18:04:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1389 2024-04-24T18:04:09.192 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1391 2024-04-24T18:04:14.211 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1393 2024-04-24T18:04:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1395 2024-04-24T18:04:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1397 2024-04-24T18:04:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1399 2024-04-24T18:04:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1401 2024-04-24T18:04:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1403 2024-04-24T18:04:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1405 2024-04-24T18:04:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1407 2024-04-24T18:04:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:04:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:04:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:04:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:04:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:04:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:04:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:04:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1409 2024-04-24T18:04:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:07:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T18:07:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T18:07:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 67527 2024-04-24T18:07:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T18:07:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T18:07:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T18:07:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=bada63500c56b0c570834289fb7d5918 2024-04-24T18:07:50.411 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T18:07:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T18:07:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:07:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:07:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:07:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T18:07:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T18:07:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T18:07:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:07:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:07:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:07:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:07:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:07:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:07:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:07:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:07:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:07:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:07:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:07:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:07:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:07:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:07:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:07:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:07:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:07:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+stopped 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: local image is primary 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:07:51 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:07:53' 2024-04-24T18:07:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:07:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 1 2024-04-24T18:07:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:07:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 2 2024-04-24T18:07:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:07:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:07:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:07:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:07:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:07:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:07:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:07:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:07:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:07:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T18:07:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:07:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:07:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:07:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:07:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:07:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:07:56 2024-04-24T18:07:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:07:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T18:07:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:07:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:07:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:07:56' 2024-04-24T18:07:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:07:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T18:07:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T18:07:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T18:07:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T18:07:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:07:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T18:07:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:07:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T18:07:57.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T18:07:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T18:07:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T18:07:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:07:56' 2024-04-24T18:07:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 18:07:56' 2024-04-24T18:07:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:07:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:07:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:07:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:07:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 18:07:56' 2024-04-24T18:07:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:07:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:07:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T18:07:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 18:07:56cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 18:07:56' 2024-04-24T18:07:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T18:07:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T18:07:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:07:56' 2024-04-24T18:07:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T18:07:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T18:07:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:07:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T18:07:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:07:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:07:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:07:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:07:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:07:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:07:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:07:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:07:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:07:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:07:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:07:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T18:07:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:07:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:07:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:07:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:07:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:07:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:07:56 2024-04-24T18:07:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:07:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T18:07:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:07:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:07:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:07:56' 2024-04-24T18:07:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:07:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T18:07:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T18:07:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T18:07:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T18:07:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:07:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T18:07:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:07:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T18:07:58.356 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T18:07:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T18:07:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T18:07:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:07:56' 2024-04-24T18:07:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 18:07:56' 2024-04-24T18:07:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:07:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:07:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:07:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:07:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 18:07:56' 2024-04-24T18:07:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:07:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:07:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T18:07:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 18:07:56cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 18:07:56' 2024-04-24T18:07:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T18:07:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T18:07:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:07:56' 2024-04-24T18:07:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T18:07:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T18:07:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:07:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T18:07:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:07:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:07:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:07:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T18:07:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:07:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:07:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:07:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T18:07:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T18:07:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T18:07:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T18:07:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T18:07:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T18:07:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T18:07:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T18:08:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=bada63500c56b0c570834289fb7d5918 2024-04-24T18:08:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ bada63500c56b0c570834289fb7d5918 != \b\a\d\a\6\3\5\0\0\c\5\6\b\0\c\5\7\0\8\3\4\2\8\9\f\b\7\d\5\9\1\8 ]] 2024-04-24T18:08:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T18:08:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T18:08:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T18:08:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T18:08:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T18:08:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T18:08:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T18:08:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T18:08:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=68568 2024-04-24T18:08:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T18:08:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T18:08:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T18:08:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T18:08:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:08:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:08:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:08:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:08:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:08:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:08:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:08:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:08:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:08:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T18:08:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T18:08:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1413 2024-04-24T18:08:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:08:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:08:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:08:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:08:25.727 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1415 2024-04-24T18:08:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:08:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:08:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:08:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:08:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:08:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:08:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:08:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1417 2024-04-24T18:08:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:08:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:08:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:08:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1419 2024-04-24T18:08:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:08:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:08:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:08:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:08:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:08:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:08:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:08:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1421 2024-04-24T18:08:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:08:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:08:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:08:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:08:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:08:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:08:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:08:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1423 2024-04-24T18:08:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:08:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:08:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:08:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:08:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:08:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:08:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:08:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1425 2024-04-24T18:08:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:08:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:08:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1427 2024-04-24T18:08:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:08:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:08:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:08:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:08:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:08:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:08:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:08:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1429 2024-04-24T18:08:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1431 2024-04-24T18:09:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1433 2024-04-24T18:09:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:12.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:14.473 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1435 2024-04-24T18:09:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1437 2024-04-24T18:09:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:22.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:24.472 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1439 2024-04-24T18:09:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1441 2024-04-24T18:09:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1443 2024-04-24T18:09:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1445 2024-04-24T18:09:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1447 2024-04-24T18:09:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1449 2024-04-24T18:09:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1451 2024-04-24T18:09:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:09:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:09:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:09:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:09:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:09:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:09:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:09:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1453 2024-04-24T18:09:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:10:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:10:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:10:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:10:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:10:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:10:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:10:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1455 2024-04-24T18:10:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:10:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:10:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:10:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:10:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:10:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:10:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:10:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1457 2024-04-24T18:10:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:10:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:10:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:10:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:10:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:10:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:10:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:10:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1459 2024-04-24T18:10:14.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:10:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:10:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:10:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:10:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:10:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:10:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:10:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1461 2024-04-24T18:10:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:10:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:10:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:10:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:10:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:10:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:10:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:10:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1463 2024-04-24T18:10:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:10:27.502 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:10:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:10:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:10:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:10:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:10:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:10:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1465 2024-04-24T18:10:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:10:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:10:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:10:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:10:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:10:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:10:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:10:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1467 2024-04-24T18:10:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:10:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:10:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:10:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:10:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:10:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:10:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:10:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1469 2024-04-24T18:10:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:10:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:10:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:10:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:10:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:10:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:10:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:10:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1471 2024-04-24T18:10:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:13:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T18:13:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T18:13:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 68568 2024-04-24T18:13:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T18:13:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T18:13:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T18:13:37.627 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=5472f6a06a040b8dbb297bf2ce5c49c2 2024-04-24T18:13:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T18:13:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T18:13:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:13:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:13:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:13:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T18:13:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T18:13:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T18:13:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:13:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:13:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:13:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:13:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:13:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:13:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:13:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:13:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:13:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:13:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:13:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:13:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:13:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:13:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:13:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:13:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:13:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:13:41 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:13:41' 2024-04-24T18:13:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:13:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T18:13:43.859 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T18:13:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T18:13:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T18:13:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:13:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T18:13:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:13:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T18:13:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T18:13:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T18:13:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T18:13:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:13:41' 2024-04-24T18:13:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 18:13:41' 2024-04-24T18:13:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:13:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:13:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:13:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:13:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 18:13:41' 2024-04-24T18:13:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:13:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:13:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T18:13:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 18:13:41cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image demoted2024-04-24 18:13:41' 2024-04-24T18:13:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T18:13:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T18:13:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:13:41' 2024-04-24T18:13:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T18:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T18:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T18:13:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:13:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:13:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:13:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:13:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:13:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:13:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:13:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:13:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:13:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:13:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:13:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T18:13:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:13:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:13:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:13:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T18:13:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:13:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:13:41 2024-04-24T18:13:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:13:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T18:13:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:13:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:13:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:13:41' 2024-04-24T18:13:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:13:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T18:13:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T18:13:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image demoted 2024-04-24T18:13:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T18:13:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:13:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T18:13:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:13:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T18:13:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T18:13:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T18:13:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T18:13:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:13:41' 2024-04-24T18:13:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 18:13:41' 2024-04-24T18:13:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:13:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:13:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:13:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:13:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 18:13:41' 2024-04-24T18:13:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:13:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:13:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T18:13:45.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image demoted42594265mirror.0smithi1362024-04-24 18:13:41cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 18:13:41' 2024-04-24T18:13:45.067 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T18:13:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T18:13:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:13:41' 2024-04-24T18:13:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T18:13:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T18:13:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:13:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T18:13:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:13:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:13:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:13:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T18:13:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:13:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:13:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:13:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T18:13:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T18:13:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T18:13:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T18:13:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T18:13:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T18:13:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T18:13:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T18:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=5472f6a06a040b8dbb297bf2ce5c49c2 2024-04-24T18:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 5472f6a06a040b8dbb297bf2ce5c49c2 != \5\4\7\2\f\6\a\0\6\a\0\4\0\b\8\d\b\b\2\9\7\b\f\2\c\e\5\c\4\9\c\2 ]] 2024-04-24T18:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T18:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T18:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T18:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T18:14:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T18:14:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T18:14:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T18:14:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T18:14:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=69565 2024-04-24T18:14:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T18:14:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster2 mirror image-alternate-primary 2024-04-24T18:14:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T18:14:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T18:14:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T18:14:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T18:14:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1479 2024-04-24T18:14:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:14:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1481 2024-04-24T18:14:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:14:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1483 2024-04-24T18:14:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:14:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1485 2024-04-24T18:14:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:14:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1487 2024-04-24T18:14:29.130 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:14:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1489 2024-04-24T18:14:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:14:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1491 2024-04-24T18:14:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:14:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:42.174 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1493 2024-04-24T18:14:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:14:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1495 2024-04-24T18:14:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:14:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1497 2024-04-24T18:14:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:14:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:14:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:14:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:14:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:14:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:14:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:14:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1499 2024-04-24T18:14:59.227 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1501 2024-04-24T18:15:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1503 2024-04-24T18:15:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1505 2024-04-24T18:15:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1507 2024-04-24T18:15:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1509 2024-04-24T18:15:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1511 2024-04-24T18:15:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1513 2024-04-24T18:15:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1515 2024-04-24T18:15:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1517 2024-04-24T18:15:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1519 2024-04-24T18:15:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1521 2024-04-24T18:15:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:15:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:15:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:15:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:15:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:15:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:15:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:15:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1523 2024-04-24T18:15:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:16:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:16:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:16:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:16:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:16:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:16:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:16:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1525 2024-04-24T18:16:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:16:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:16:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:16:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:16:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:16:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:16:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:16:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1527 2024-04-24T18:16:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:16:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:16:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:16:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:16:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:16:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:16:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:16:14.414 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1529 2024-04-24T18:16:14.426 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:16:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:16:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:16:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:16:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:16:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:16:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:16:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1531 2024-04-24T18:16:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:16:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:16:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:16:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:16:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:16:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:16:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:16:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1533 2024-04-24T18:16:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:16:27.447 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:16:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:16:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:16:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:16:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:16:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:16:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1535 2024-04-24T18:16:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:16:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:16:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster2 mirror image-alternate-primary 2024-04-24T18:16:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:16:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:16:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:16:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:16:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1537 2024-04-24T18:16:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:19:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T18:19:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T18:19:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 69565 2024-04-24T18:19:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T18:19:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T18:19:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T18:19:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=4ab4dc38073dcc5a752d98febe6aa453 2024-04-24T18:19:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-04-24T18:19:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster2 mirror image-alternate-primary 2024-04-24T18:19:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:19:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/image-alternate-primary 2024-04-24T18:19:30.257 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T18:19:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T18:19:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:19:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:19:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:19:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:19:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:19:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:19:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:19:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:19:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:19:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:19:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:19:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:19:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:19:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T18:19:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:19:31 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:19:31' 2024-04-24T18:19:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:19:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T18:19:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T18:19:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T18:19:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T18:19:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:19:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T18:19:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:19:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T18:19:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T18:19:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T18:19:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T18:19:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:19:31' 2024-04-24T18:19:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 18:19:31' 2024-04-24T18:19:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:19:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:19:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 18:19:31' 2024-04-24T18:19:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:19:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:19:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T18:19:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 18:19:31cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 18:19:31' 2024-04-24T18:19:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T18:19:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T18:19:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:19:31' 2024-04-24T18:19:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T18:19:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T18:19:31.492 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:19:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T18:19:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:19:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:19:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:19:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T18:19:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:19:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:31.494 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:19:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:19:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:19:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:19:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:19:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:19:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:19:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:19:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:19:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T18:19:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:19:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:19:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:19:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:19:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:19:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:19:31 2024-04-24T18:19:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:19:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T18:19:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:19:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:19:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:19:31' 2024-04-24T18:19:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:19:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T18:19:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T18:19:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T18:19:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T18:19:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:19:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T18:19:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:19:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T18:19:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T18:19:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T18:19:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T18:19:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:19:31' 2024-04-24T18:19:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 18:19:31' 2024-04-24T18:19:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:19:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:19:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 18:19:31' 2024-04-24T18:19:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:19:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:19:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T18:19:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 18:19:31cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 18:19:31' 2024-04-24T18:19:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T18:19:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T18:19:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:19:31' 2024-04-24T18:19:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T18:19:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T18:19:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:19:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T18:19:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:19:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:19:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:19:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster1 mirror image-alternate-primary 2024-04-24T18:19:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:19:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T18:19:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/image-alternate-primary 2024-04-24T18:19:32.894 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T18:19:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T18:19:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T18:19:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T18:19:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T18:19:33.205 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T18:19:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=4ab4dc38073dcc5a752d98febe6aa453 2024-04-24T18:19:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ 4ab4dc38073dcc5a752d98febe6aa453 != \4\a\b\4\d\c\3\8\0\7\3\d\c\c\5\a\7\5\2\d\9\8\f\e\b\e\6\a\a\4\5\3 ]] 2024-04-24T18:19:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster2 2024-04-24T18:19:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster1 2024-04-24T18:19:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster2 2024-04-24T18:19:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..25} 2024-04-24T18:19:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo mount /dev/nbd0 test-alternate-primary 2024-04-24T18:19:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ whoami 2024-04-24T18:19:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo chown ubuntu test-alternate-primary 2024-04-24T18:19:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -rf test-alternate-primary/linux-5.4 test-alternate-primary/linux-5.4.tar.gz 2024-04-24T18:19:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ SNAP_PID=70602 2024-04-24T18:19:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ slow_untar_workload test-alternate-primary 2024-04-24T18:19:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local mountpt=test-alternate-primary 2024-04-24T18:19:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ take_mirror_snapshots cluster1 mirror image-alternate-primary 2024-04-24T18:19:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cp linux-5.4.tar.gz test-alternate-primary 2024-04-24T18:19:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:19:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:19:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:19:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:19:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:19:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local ret=0 2024-04-24T18:19:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ timeout 5m bash -c 'zcat test-alternate-primary/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-alternate-primary' 2024-04-24T18:19:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1541 2024-04-24T18:19:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:19:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:19:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:19:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:19:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:19:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:19:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:19:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1543 2024-04-24T18:19:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1545 2024-04-24T18:20:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1547 2024-04-24T18:20:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1549 2024-04-24T18:20:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1551 2024-04-24T18:20:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:21.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1553 2024-04-24T18:20:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1555 2024-04-24T18:20:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1557 2024-04-24T18:20:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1559 2024-04-24T18:20:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1561 2024-04-24T18:20:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1563 2024-04-24T18:20:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1565 2024-04-24T18:20:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:20:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:20:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:20:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:20:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:20:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:20:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:20:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1567 2024-04-24T18:20:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:01.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1569 2024-04-24T18:21:03.596 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1571 2024-04-24T18:21:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1573 2024-04-24T18:21:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1575 2024-04-24T18:21:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1577 2024-04-24T18:21:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1579 2024-04-24T18:21:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1581 2024-04-24T18:21:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1583 2024-04-24T18:21:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1585 2024-04-24T18:21:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1587 2024-04-24T18:21:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:53.596 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1589 2024-04-24T18:21:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:21:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:21:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:21:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:21:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:21:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:21:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:21:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1591 2024-04-24T18:21:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:22:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:22:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:22:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:22:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:22:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:22:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:22:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1593 2024-04-24T18:22:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:22:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:22:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:22:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:22:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:22:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:22:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:22:08.613 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1595 2024-04-24T18:22:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:22:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:22:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:22:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:22:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:22:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:22:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:22:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1597 2024-04-24T18:22:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:22:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for i in {1..30} 2024-04-24T18:22:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ mirror_image_snapshot cluster1 mirror image-alternate-primary 2024-04-24T18:22:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:22:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:22:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:22:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-alternate-primary 2024-04-24T18:22:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Snapshot ID: 1599 2024-04-24T18:22:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 3 2024-04-24T18:24:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ret=124 2024-04-24T18:24:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ (( ret != 124 )) 2024-04-24T18:24:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait 70602 2024-04-24T18:24:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo umount test-alternate-primary 2024-04-24T18:24:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T18:24:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T18:25:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEMOTE_MD5=f0d0711058409138edb32c283b69434a 2024-04-24T18:25:11.030 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd /dev/nbd0 2024-04-24T18:25:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ demote_image cluster1 mirror image-alternate-primary 2024-04-24T18:25:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:25:12.101 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:25:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:25:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-alternate-primary 2024-04-24T18:25:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image demoted to non-primary 2024-04-24T18:25:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown 2024-04-24T18:25:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:25:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:25:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:25:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:25:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:25:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:25:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:25:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:25:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:25:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:25:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:25:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:25:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:25:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:25:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:25:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:25:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:25:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T18:25:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:25:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:25:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+stopped 2024-04-24T18:25:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: local image is primary 2024-04-24T18:25:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:25:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:25:11 2024-04-24T18:25:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:25:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T18:25:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:25:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image demoted 2024-04-24T18:25:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:25:14' 2024-04-24T18:25:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:25:16.053 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 1 2024-04-24T18:25:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:25:16.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 2 2024-04-24T18:25:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster1 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:25:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:25:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:25:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:25:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:25:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:25:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:25:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:25:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:25:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-alternate-primary 2024-04-24T18:25:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:25:16 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster2 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:25:16' 2024-04-24T18:25:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:25:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T18:25:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T18:25:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T18:25:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T18:25:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:25:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T18:25:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:25:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T18:25:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T18:25:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T18:25:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T18:25:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:25:16' 2024-04-24T18:25:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-alternate-primary up+unknown '2024-04-24 18:25:16' 2024-04-24T18:25:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:25:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:25:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:25:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:25:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 18:25:16' 2024-04-24T18:25:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:25:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:25:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-24T18:25:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142614264mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42614264mirror.0smithi1362024-04-24 18:25:16cluster2bbe878a2-79b9-47b5-8f47-bbbe1aa0fa71up+unknownremote image is not primary2024-04-24 18:25:16' 2024-04-24T18:25:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T18:25:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T18:25:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:25:16' 2024-04-24T18:25:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T18:25:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T18:25:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:25:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T18:25:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:25:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:25:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:25:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown 2024-04-24T18:25:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:25:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:25:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:25:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:25:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:25:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:25:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-24T18:25:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:25:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_status_in_pool_dir cluster2 mirror image-alternate-primary up+unknown '' '' 2024-04-24T18:25:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:25:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:25:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:25:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:25:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local description_pattern= 2024-04-24T18:25:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local service_pattern= 2024-04-24T18:25:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:25:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:25:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-alternate-primary 2024-04-24T18:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='image-alternate-primary: 2024-04-24T18:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: global_id: 78d16678-be33-4652-b10d-f56d44f08914 2024-04-24T18:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: service: mirror.0 on smithi136 2024-04-24T18:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:25:16 2024-04-24T18:25:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: peer_sites: 2024-04-24T18:25:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: name: cluster1 2024-04-24T18:25:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: state: up+unknown 2024-04-24T18:25:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: description: remote image is not primary 2024-04-24T18:25:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr: last_update: 2024-04-24 18:25:16' 2024-04-24T18:25:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ state: .*up+unknown' 2024-04-24T18:25:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: state: up+unknown 2024-04-24T18:25:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^ description: .*' 2024-04-24T18:25:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: description: remote image is not primary 2024-04-24T18:25:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n '' ']' 2024-04-24T18:25:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:25:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep '^up+' 2024-04-24T18:25:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:25:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep 'service: *mirror..* on ' 2024-04-24T18:25:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout: service: mirror.0 on smithi136 2024-04-24T18:25:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update 2024-04-24T18:25:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-24T18:25:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:25:16' 2024-04-24T18:25:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-alternate-primary up+unknown '2024-04-24 18:25:16' 2024-04-24T18:25:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:25:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:25:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:25:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local state_pattern=up+unknown 2024-04-24T18:25:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local 'prev_last_update=2024-04-24 18:25:16' 2024-04-24T18:25:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local status 2024-04-24T18:25:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ CEPH_ARGS= 2024-04-24T18:25:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-24T18:25:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ status='WARNINGOKWARNING142594265mirror.0smithi13617.2.7-915-g330067dbtrueOKimage-alternate-primary78d16678-be33-4652-b10d-f56d44f08914up+unknownremote image is not primary42594265mirror.0smithi1362024-04-24 18:25:16cluster1c2a51700-2fab-41de-a597-ea1b2da28c94up+unknownremote image is not primary2024-04-24 18:25:16' 2024-04-24T18:25:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local last_update state 2024-04-24T18:25:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/last_update' 2024-04-24T18:25:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ last_update='2024-04-24 18:25:16' 2024-04-24T18:25:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-alternate-primary'\'']/state' 2024-04-24T18:25:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ state=up+unknown 2024-04-24T18:25:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo up+unknown 2024-04-24T18:25:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ grep up+unknown 2024-04-24T18:25:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:up+unknown 2024-04-24T18:25:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ return 0 2024-04-24T18:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ promote_image cluster2 mirror image-alternate-primary 2024-04-24T18:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pool=mirror 2024-04-24T18:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local image=image-alternate-primary 2024-04-24T18:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local force= 2024-04-24T18:25:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-alternate-primary 2024-04-24T18:25:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:Image promoted to primary 2024-04-24T18:25:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ nbd == \n\b\d ]] 2024-04-24T18:25:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-alternate-primary 2024-04-24T18:25:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ DEV=/dev/nbd0 2024-04-24T18:25:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ sudo md5sum /dev/nbd0 2024-04-24T18:25:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ awk '{print $1}' 2024-04-24T18:25:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ PROMOTE_MD5=f0d0711058409138edb32c283b69434a 2024-04-24T18:25:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ [[ f0d0711058409138edb32c283b69434a != \f\0\d\0\7\1\1\0\5\8\4\0\9\1\3\8\e\d\b\3\2\c\2\8\3\b\6\9\4\3\4\a ]] 2024-04-24T18:25:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ TEMP=cluster1 2024-04-24T18:25:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER1=cluster2 2024-04-24T18:25:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CLUSTER2=cluster1 2024-04-24T18:25:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo OK 2024-04-24T18:25:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:OK 2024-04-24T18:25:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ cleanup 0 2024-04-24T18:25:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local error_code=0 2024-04-24T18:25:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ set +e 2024-04-24T18:25:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' 0 -ne 0 ']' 2024-04-24T18:25:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -z '' ']' 2024-04-24T18:25:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster instance 2024-04-24T18:25:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T18:25:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-24T18:25:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:pool 'mirror' does not exist 2024-04-24T18:25:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T18:25:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-24T18:25:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:pool 'mirror' does not exist 2024-04-24T18:25:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T18:25:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-24T18:25:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:pool 'mirror_parent' does not exist 2024-04-24T18:25:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ CEPH_ARGS= 2024-04-24T18:25:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-24T18:25:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:pool 'mirror_parent' does not exist 2024-04-24T18:25:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-24T18:25:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ stop_mirrors cluster2 2024-04-24T18:25:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2 2024-04-24T18:25:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local sig= 2024-04-24T18:25:45.645 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ seq 0 0 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ stop_mirror cluster2:0 '' 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster2:0 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local sig= 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test -n '' 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pid 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ daemon_pid_file cluster2:0 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local cluster=cluster2:0 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local instance 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-04-24T18:25:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local val=cluster2:0 2024-04-24T18:25:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local cluster_var_name=cluster 2024-04-24T18:25:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local instance_var_name=instance 2024-04-24T18:25:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ cluster=cluster2 2024-04-24T18:25:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ instance=0 2024-04-24T18:25:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ '[' 0 = cluster2:0 ']' 2024-04-24T18:25:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ eval cluster=cluster2 2024-04-24T18:25:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++++ cluster=cluster2 2024-04-24T18:25:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ eval instance=0 2024-04-24T18:25:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++++ instance=0 2024-04-24T18:25:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-04-24T18:25:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-04-24T18:25:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-04-24T18:25:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ pid=45974 2024-04-24T18:25:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n 45974 ']' 2024-04-24T18:25:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ kill 45974 2024-04-24T18:25:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T18:25:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:25:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ps auxww 2024-04-24T18:25:46.763 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ awk -v pid=45974 '$2 == pid {print; exit 1}' 2024-04-24T18:25:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ break 2024-04-24T18:25:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ps auxww 2024-04-24T18:25:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ awk -v pid=45974 '$2 == pid {print; exit 1}' 2024-04-24T18:25:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-04-24T18:25:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local local_cluster=cluster2:0 2024-04-24T18:25:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster=cluster2 2024-04-24T18:25:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance 2024-04-24T18:25:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-04-24T18:25:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local val=cluster2:0 2024-04-24T18:25:46.824 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster_var_name=local_cluster 2024-04-24T18:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance_var_name=instance 2024-04-24T18:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ cluster=cluster2 2024-04-24T18:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ instance=0 2024-04-24T18:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ '[' 0 = cluster2:0 ']' 2024-04-24T18:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval local_cluster=cluster2 2024-04-24T18:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local_cluster=cluster2 2024-04-24T18:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval instance=0 2024-04-24T18:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ instance=0 2024-04-24T18:25:46.825 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-04-24T18:25:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-04-24T18:25:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local local_cluster=cluster2:0 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster=cluster1 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local val=cluster2:0 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster_var_name=local_cluster 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance_var_name=instance 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ cluster=cluster2 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ instance=0 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ '[' 0 = cluster2:0 ']' 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval local_cluster=cluster2 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local_cluster=cluster2 2024-04-24T18:25:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval instance=0 2024-04-24T18:25:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ instance=0 2024-04-24T18:25:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-04-24T18:25:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-04-24T18:25:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-04-24T18:25:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ daemon_pid_file cluster2:0 2024-04-24T18:25:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster=cluster2:0 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local val=cluster2:0 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster_var_name=cluster 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance_var_name=instance 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ cluster=cluster2 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ instance=0 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ '[' 0 = cluster2:0 ']' 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval cluster=cluster2 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ cluster=cluster2 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval instance=0 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ instance=0 2024-04-24T18:25:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-04-24T18:25:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-04-24T18:25:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-04-24T18:25:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-24T18:25:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ stop_mirrors cluster1 2024-04-24T18:25:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1 2024-04-24T18:25:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local sig= 2024-04-24T18:25:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ seq 0 0 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ stop_mirror cluster1:0 '' 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local cluster=cluster1:0 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local sig= 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test -n '' 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ local pid 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ daemon_pid_file cluster1:0 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local cluster=cluster1:0 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local instance 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local val=cluster1:0 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local cluster_var_name=cluster 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local instance_var_name=instance 2024-04-24T18:25:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ cluster=cluster1 2024-04-24T18:25:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ instance=0 2024-04-24T18:25:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ '[' 0 = cluster1:0 ']' 2024-04-24T18:25:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ eval cluster=cluster1 2024-04-24T18:25:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++++ cluster=cluster1 2024-04-24T18:25:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ eval instance=0 2024-04-24T18:25:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++++ instance=0 2024-04-24T18:25:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-04-24T18:25:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-04-24T18:25:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-04-24T18:25:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ pid=45945 2024-04-24T18:25:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -n 45945 ']' 2024-04-24T18:25:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ kill 45945 2024-04-24T18:25:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ for s in 1 2 4 8 16 32 2024-04-24T18:25:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ sleep 1 2024-04-24T18:25:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ps auxww 2024-04-24T18:25:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ awk -v pid=45945 '$2 == pid {print; exit 1}' 2024-04-24T18:25:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ break 2024-04-24T18:25:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ ps auxww 2024-04-24T18:25:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ awk -v pid=45945 '$2 == pid {print; exit 1}' 2024-04-24T18:25:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local local_cluster=cluster1:0 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster=cluster2 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local val=cluster1:0 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster_var_name=local_cluster 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance_var_name=instance 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ cluster=cluster1 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ instance=0 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval local_cluster=cluster1 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local_cluster=cluster1 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval instance=0 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ instance=0 2024-04-24T18:25:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T18:25:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T18:25:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T18:25:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local local_cluster=cluster1:0 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster=cluster1 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local val=cluster1:0 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster_var_name=local_cluster 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance_var_name=instance 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ cluster=cluster1 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ instance=0 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval local_cluster=cluster1 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ local_cluster=cluster1 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval instance=0 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ instance=0 2024-04-24T18:25:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-04-24T18:25:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T18:25:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-04-24T18:25:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ daemon_pid_file cluster1:0 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster=cluster1:0 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local val=cluster1:0 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local cluster_var_name=cluster 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ local instance_var_name=instance 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ cluster=cluster1 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ instance=0 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ '[' 0 = cluster1:0 ']' 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval cluster=cluster1 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ cluster=cluster1 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ eval instance=0 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ instance=0 2024-04-24T18:25:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-04-24T18:25:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-04-24T18:25:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-04-24T18:25:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' -z 1 ']' 2024-04-24T18:25:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ test '' = /tmp/tmp.gEigttWnJf 2024-04-24T18:25:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ rm -Rf /tmp/tmp.gEigttWnJf 2024-04-24T18:25:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ '[' 0 -eq 0 ']' 2024-04-24T18:25:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stdout:OK 2024-04-24T18:25:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ echo OK 2024-04-24T18:25:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi136.stderr:+ exit 0 2024-04-24T18:25:47.989 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-24T18:25:47.990 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-04-24T18:25:48.127 INFO:tasks.workunit:Stopping ['rbd/compare_mirror_image_alternate_primary.sh'] on cluster1.client.mirror... 2024-04-24T18:25:48.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-04-24T18:25:48.380 DEBUG:teuthology.parallel:result is None 2024-04-24T18:25:48.381 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-24T18:25:48.407 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-24T18:25:48.407 DEBUG:teuthology.orchestra.run.smithi136:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-24T18:25:48.464 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-24T18:25:48.465 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T18:25:48.475 INFO:tasks.ceph.ceph_manager.cluster2:Canceling any pending splits or merges... 2024-04-24T18:25:48.475 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T18:25:48.821 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T18:25:48.821 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":2154,"fsid":"e6ccc4a4-622a-400f-a916-d56c405e9e43","created":"2024-04-24T15:59:33.767454+0000","modified":"2024-04-24T18:25:44.910351+0000","last_up_change":"2024-04-24T15:59:44.747907+0000","last_in_change":"2024-04-24T15:59:34.715818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T15:59:45.382463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T15:59:49.063456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cfb39152-2b32-4f85-9b4a-f2ffcc9fed42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":40663},{"type":"v1","addr":"172.21.15.136:6802","nonce":40663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":40663},{"type":"v1","addr":"172.21.15.136:6806","nonce":40663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":40663},{"type":"v1","addr":"172.21.15.136:6814","nonce":40663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":40663},{"type":"v1","addr":"172.21.15.136:6810","nonce":40663}]},"public_addr":"172.21.15.136:6802/40663","cluster_addr":"172.21.15.136:6806/40663","heartbeat_back_addr":"172.21.15.136:6814/40663","heartbeat_front_addr":"172.21.15.136:6810/40663","state":["exists","up"]},{"osd":1,"uuid":"80d9ea77-6b87-44c8-bf04-a95b22ad41f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":40665},{"type":"v1","addr":"172.21.15.136:6817","nonce":40665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":40665},{"type":"v1","addr":"172.21.15.136:6819","nonce":40665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":40665},{"type":"v1","addr":"172.21.15.136:6823","nonce":40665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":40665},{"type":"v1","addr":"172.21.15.136:6821","nonce":40665}]},"public_addr":"172.21.15.136:6817/40665","cluster_addr":"172.21.15.136:6819/40665","heartbeat_back_addr":"172.21.15.136:6823/40665","heartbeat_front_addr":"172.21.15.136:6821/40665","state":["exists","up"]},{"osd":2,"uuid":"240de483-dd6c-4282-ad4c-f5fdb8a32545","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":40664},{"type":"v1","addr":"172.21.15.136:6803","nonce":40664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":40664},{"type":"v1","addr":"172.21.15.136:6807","nonce":40664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":40664},{"type":"v1","addr":"172.21.15.136:6815","nonce":40664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":40664},{"type":"v1","addr":"172.21.15.136:6811","nonce":40664}]},"public_addr":"172.21.15.136:6803/40664","cluster_addr":"172.21.15.136:6807/40664","heartbeat_back_addr":"172.21.15.136:6815/40664","heartbeat_front_addr":"172.21.15.136:6811/40664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.824415+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.805861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.799092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T18:25:48.835 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-24T18:25:48.836 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T18:25:49.145 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T18:25:49.147 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-24T18:25:49.158 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":6532,"stamp":"2024-04-24T18:25:47.789015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":919824,"kb_used_data":14748,"kb_used_omap":4,"kb_used_meta":904315,"kb_avail":282195696,"statfs":{"total":289910292480,"available":288968392704,"internally_reserved":0,"allocated":15101952,"data_stored":9270544,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":926018568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1042410375,"num_objects":-310,"num_object_clones":-101,"num_object_copies":-620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-310,"num_whiteouts":0,"num_read":-464729,"num_read_kb":-21480062,"num_write":-36737,"num_write_kb":-8493308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-12738579,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001789"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.956268+0000","last_change":"2024-04-24T15:59:50.776460+0000","last_active":"2024-04-24T18:25:42.956268+0000","last_peered":"2024-04-24T18:25:42.956268+0000","last_clean":"2024-04-24T18:25:42.956268+0000","last_became_active":"2024-04-24T15:59:50.776173+0000","last_became_peered":"2024-04-24T15:59:50.776173+0000","last_unstale":"2024-04-24T18:25:42.956268+0000","last_undegraded":"2024-04-24T18:25:42.956268+0000","last_fullsized":"2024-04-24T18:25:42.956268+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:54:40.948714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.579867+0000","last_change":"2024-04-24T15:59:50.774230+0000","last_active":"2024-04-24T18:25:43.579867+0000","last_peered":"2024-04-24T18:25:43.579867+0000","last_clean":"2024-04-24T18:25:43.579867+0000","last_became_active":"2024-04-24T15:59:50.773938+0000","last_became_peered":"2024-04-24T15:59:50.773938+0000","last_unstale":"2024-04-24T18:25:43.579867+0000","last_undegraded":"2024-04-24T18:25:43.579867+0000","last_fullsized":"2024-04-24T18:25:43.579867+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:54.764334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2145,"reported_epoch":2152,"state":"active+clean","last_fresh":"2024-04-24T18:25:20.143238+0000","last_change":"2024-04-24T15:59:50.775383+0000","last_active":"2024-04-24T18:25:20.143238+0000","last_peered":"2024-04-24T18:25:20.143238+0000","last_clean":"2024-04-24T18:25:20.143238+0000","last_became_active":"2024-04-24T15:59:50.775119+0000","last_became_peered":"2024-04-24T15:59:50.775119+0000","last_unstale":"2024-04-24T18:25:20.143238+0000","last_undegraded":"2024-04-24T18:25:20.143238+0000","last_fullsized":"2024-04-24T18:25:20.143238+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:20:11.243695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.953534+0000","last_change":"2024-04-24T15:59:50.775216+0000","last_active":"2024-04-24T18:25:42.953534+0000","last_peered":"2024-04-24T18:25:42.953534+0000","last_clean":"2024-04-24T18:25:42.953534+0000","last_became_active":"2024-04-24T15:59:50.774935+0000","last_became_peered":"2024-04-24T15:59:50.774935+0000","last_unstale":"2024-04-24T18:25:42.953534+0000","last_undegraded":"2024-04-24T18:25:42.953534+0000","last_fullsized":"2024-04-24T18:25:42.953534+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:50.389462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":2276,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.948563+0000","last_change":"2024-04-24T15:59:46.916688+0000","last_active":"2024-04-24T18:25:42.948563+0000","last_peered":"2024-04-24T18:25:42.948563+0000","last_clean":"2024-04-24T18:25:42.948563+0000","last_became_active":"2024-04-24T15:59:46.915637+0000","last_became_peered":"2024-04-24T15:59:46.915637+0000","last_unstale":"2024-04-24T18:25:42.948563+0000","last_undegraded":"2024-04-24T18:25:42.948563+0000","last_fullsized":"2024-04-24T18:25:42.948563+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_clean_scrub_stamp":"2024-04-24T15:59:45.750019+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:30:29.745322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2145,"reported_epoch":2152,"state":"active+clean","last_fresh":"2024-04-24T18:25:20.143006+0000","last_change":"2024-04-24T15:59:50.775501+0000","last_active":"2024-04-24T18:25:20.143006+0000","last_peered":"2024-04-24T18:25:20.143006+0000","last_clean":"2024-04-24T18:25:20.143006+0000","last_became_active":"2024-04-24T15:59:50.775157+0000","last_became_peered":"2024-04-24T15:59:50.775157+0000","last_unstale":"2024-04-24T18:25:20.143006+0000","last_undegraded":"2024-04-24T18:25:20.143006+0000","last_fullsized":"2024-04-24T18:25:20.143006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:43:00.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.956227+0000","last_change":"2024-04-24T15:59:50.777598+0000","last_active":"2024-04-24T18:25:42.956227+0000","last_peered":"2024-04-24T18:25:42.956227+0000","last_clean":"2024-04-24T18:25:42.956227+0000","last_became_active":"2024-04-24T15:59:50.777322+0000","last_became_peered":"2024-04-24T15:59:50.777322+0000","last_unstale":"2024-04-24T18:25:42.956227+0000","last_undegraded":"2024-04-24T18:25:42.956227+0000","last_fullsized":"2024-04-24T18:25:42.956227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:54:24.332122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.962339+0000","last_change":"2024-04-24T15:59:50.774597+0000","last_active":"2024-04-24T18:25:42.962339+0000","last_peered":"2024-04-24T18:25:42.962339+0000","last_clean":"2024-04-24T18:25:42.962339+0000","last_became_active":"2024-04-24T15:59:50.774160+0000","last_became_peered":"2024-04-24T15:59:50.774160+0000","last_unstale":"2024-04-24T18:25:42.962339+0000","last_undegraded":"2024-04-24T18:25:42.962339+0000","last_fullsized":"2024-04-24T18:25:42.962339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:12.412466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.960306+0000","last_change":"2024-04-24T15:59:50.776430+0000","last_active":"2024-04-24T18:25:42.960306+0000","last_peered":"2024-04-24T18:25:42.960306+0000","last_clean":"2024-04-24T18:25:42.960306+0000","last_became_active":"2024-04-24T15:59:50.776148+0000","last_became_peered":"2024-04-24T15:59:50.776148+0000","last_unstale":"2024-04-24T18:25:42.960306+0000","last_undegraded":"2024-04-24T18:25:42.960306+0000","last_fullsized":"2024-04-24T18:25:42.960306+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:30:20.365824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949674719,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302528,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069312,"statfs":{"total":96636764160,"available":96326975488,"internally_reserved":0,"allocated":5050368,"data_stored":3286958,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"min":{"1min":0.20799999999999999,"5min":0.20100000000000001,"15min":0.17499999999999999},"max":{"1min":1.722,"5min":4.758,"15min":4.758},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.442,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":1.768,"5min":4.7919999999999998,"15min":4.7919999999999998},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Apr 24 18:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.34499999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.182},"max":{"1min":2.0680000000000001,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":0.214},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.42699999999999999,"15min":0.38700000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":0.53500000000000003,"5min":4.1790000000000003,"15min":4.1790000000000003},"last":0.25}]}]},{"osd":0,"up_from":9,"seq":38654707429,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314820,"kb_used_data":4936,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94057020,"statfs":{"total":96636764160,"available":96314388480,"internally_reserved":0,"allocated":5054464,"data_stored":3286977,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.26400000000000001,"5min":0.20200000000000001,"15min":0.17499999999999999},"max":{"1min":0.44500000000000001,"5min":0.46200000000000002,"15min":5.3200000000000003},"last":0.248},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.34499999999999997,"15min":0.36699999999999999},"min":{"1min":0.23899999999999999,"5min":0.222,"15min":0.17799999999999999},"max":{"1min":0.44600000000000001,"5min":0.51800000000000002,"15min":5.2039999999999997},"last":0.30299999999999999}]},{"osd":2,"last update":"Wed Apr 24 18:25:07 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34499999999999997,"15min":0.35199999999999998},"min":{"1min":0.255,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":0.46300000000000002,"5min":0.53700000000000003,"15min":4.2800000000000002},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":0.502,"5min":0.57299999999999995,"15min":7.0679999999999996},"last":0.26400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654707426,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302476,"kb_used_data":4880,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069364,"statfs":{"total":96636764160,"available":96327028736,"internally_reserved":0,"allocated":4997120,"data_stored":2696609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:38 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33300000000000002,"15min":0.35699999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":0.47199999999999998,"5min":0.55400000000000005,"15min":7.0199999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34699999999999998,"15min":0.40100000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.20699999999999999},"max":{"1min":0.752,"5min":0.752,"15min":11.084},"last":0.36499999999999999}]},{"osd":1,"last update":"Wed Apr 24 18:25:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35299999999999998,"15min":0.35999999999999999},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.216},"max":{"1min":0.46899999999999997,"5min":0.51300000000000001,"15min":5.0800000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.45000000000000001,"15min":0.38400000000000001},"min":{"1min":0.246,"5min":0.214,"15min":0.214},"max":{"1min":0.51200000000000001,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:25:49.159 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T18:25:49.470 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T18:25:49.472 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-24T18:25:49.483 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":6532,"stamp":"2024-04-24T18:25:47.789015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":919824,"kb_used_data":14748,"kb_used_omap":4,"kb_used_meta":904315,"kb_avail":282195696,"statfs":{"total":289910292480,"available":288968392704,"internally_reserved":0,"allocated":15101952,"data_stored":9270544,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":926018568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1042410375,"num_objects":-310,"num_object_clones":-101,"num_object_copies":-620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-310,"num_whiteouts":0,"num_read":-464729,"num_read_kb":-21480062,"num_write":-36737,"num_write_kb":-8493308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-12738579,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001789"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.956268+0000","last_change":"2024-04-24T15:59:50.776460+0000","last_active":"2024-04-24T18:25:42.956268+0000","last_peered":"2024-04-24T18:25:42.956268+0000","last_clean":"2024-04-24T18:25:42.956268+0000","last_became_active":"2024-04-24T15:59:50.776173+0000","last_became_peered":"2024-04-24T15:59:50.776173+0000","last_unstale":"2024-04-24T18:25:42.956268+0000","last_undegraded":"2024-04-24T18:25:42.956268+0000","last_fullsized":"2024-04-24T18:25:42.956268+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:54:40.948714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.579867+0000","last_change":"2024-04-24T15:59:50.774230+0000","last_active":"2024-04-24T18:25:43.579867+0000","last_peered":"2024-04-24T18:25:43.579867+0000","last_clean":"2024-04-24T18:25:43.579867+0000","last_became_active":"2024-04-24T15:59:50.773938+0000","last_became_peered":"2024-04-24T15:59:50.773938+0000","last_unstale":"2024-04-24T18:25:43.579867+0000","last_undegraded":"2024-04-24T18:25:43.579867+0000","last_fullsized":"2024-04-24T18:25:43.579867+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:54.764334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2145,"reported_epoch":2152,"state":"active+clean","last_fresh":"2024-04-24T18:25:20.143238+0000","last_change":"2024-04-24T15:59:50.775383+0000","last_active":"2024-04-24T18:25:20.143238+0000","last_peered":"2024-04-24T18:25:20.143238+0000","last_clean":"2024-04-24T18:25:20.143238+0000","last_became_active":"2024-04-24T15:59:50.775119+0000","last_became_peered":"2024-04-24T15:59:50.775119+0000","last_unstale":"2024-04-24T18:25:20.143238+0000","last_undegraded":"2024-04-24T18:25:20.143238+0000","last_fullsized":"2024-04-24T18:25:20.143238+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:20:11.243695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.953534+0000","last_change":"2024-04-24T15:59:50.775216+0000","last_active":"2024-04-24T18:25:42.953534+0000","last_peered":"2024-04-24T18:25:42.953534+0000","last_clean":"2024-04-24T18:25:42.953534+0000","last_became_active":"2024-04-24T15:59:50.774935+0000","last_became_peered":"2024-04-24T15:59:50.774935+0000","last_unstale":"2024-04-24T18:25:42.953534+0000","last_undegraded":"2024-04-24T18:25:42.953534+0000","last_fullsized":"2024-04-24T18:25:42.953534+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:50.389462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":2276,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.948563+0000","last_change":"2024-04-24T15:59:46.916688+0000","last_active":"2024-04-24T18:25:42.948563+0000","last_peered":"2024-04-24T18:25:42.948563+0000","last_clean":"2024-04-24T18:25:42.948563+0000","last_became_active":"2024-04-24T15:59:46.915637+0000","last_became_peered":"2024-04-24T15:59:46.915637+0000","last_unstale":"2024-04-24T18:25:42.948563+0000","last_undegraded":"2024-04-24T18:25:42.948563+0000","last_fullsized":"2024-04-24T18:25:42.948563+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_clean_scrub_stamp":"2024-04-24T15:59:45.750019+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:30:29.745322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2145,"reported_epoch":2152,"state":"active+clean","last_fresh":"2024-04-24T18:25:20.143006+0000","last_change":"2024-04-24T15:59:50.775501+0000","last_active":"2024-04-24T18:25:20.143006+0000","last_peered":"2024-04-24T18:25:20.143006+0000","last_clean":"2024-04-24T18:25:20.143006+0000","last_became_active":"2024-04-24T15:59:50.775157+0000","last_became_peered":"2024-04-24T15:59:50.775157+0000","last_unstale":"2024-04-24T18:25:20.143006+0000","last_undegraded":"2024-04-24T18:25:20.143006+0000","last_fullsized":"2024-04-24T18:25:20.143006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:43:00.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.956227+0000","last_change":"2024-04-24T15:59:50.777598+0000","last_active":"2024-04-24T18:25:42.956227+0000","last_peered":"2024-04-24T18:25:42.956227+0000","last_clean":"2024-04-24T18:25:42.956227+0000","last_became_active":"2024-04-24T15:59:50.777322+0000","last_became_peered":"2024-04-24T15:59:50.777322+0000","last_unstale":"2024-04-24T18:25:42.956227+0000","last_undegraded":"2024-04-24T18:25:42.956227+0000","last_fullsized":"2024-04-24T18:25:42.956227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:54:24.332122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.962339+0000","last_change":"2024-04-24T15:59:50.774597+0000","last_active":"2024-04-24T18:25:42.962339+0000","last_peered":"2024-04-24T18:25:42.962339+0000","last_clean":"2024-04-24T18:25:42.962339+0000","last_became_active":"2024-04-24T15:59:50.774160+0000","last_became_peered":"2024-04-24T15:59:50.774160+0000","last_unstale":"2024-04-24T18:25:42.962339+0000","last_undegraded":"2024-04-24T18:25:42.962339+0000","last_fullsized":"2024-04-24T18:25:42.962339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:12.412466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.960306+0000","last_change":"2024-04-24T15:59:50.776430+0000","last_active":"2024-04-24T18:25:42.960306+0000","last_peered":"2024-04-24T18:25:42.960306+0000","last_clean":"2024-04-24T18:25:42.960306+0000","last_became_active":"2024-04-24T15:59:50.776148+0000","last_became_peered":"2024-04-24T15:59:50.776148+0000","last_unstale":"2024-04-24T18:25:42.960306+0000","last_undegraded":"2024-04-24T18:25:42.960306+0000","last_fullsized":"2024-04-24T18:25:42.960306+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:30:20.365824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949674719,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302528,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069312,"statfs":{"total":96636764160,"available":96326975488,"internally_reserved":0,"allocated":5050368,"data_stored":3286958,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"min":{"1min":0.20799999999999999,"5min":0.20100000000000001,"15min":0.17499999999999999},"max":{"1min":1.722,"5min":4.758,"15min":4.758},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.442,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":1.768,"5min":4.7919999999999998,"15min":4.7919999999999998},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Apr 24 18:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.34499999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.182},"max":{"1min":2.0680000000000001,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":0.214},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.42699999999999999,"15min":0.38700000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":0.53500000000000003,"5min":4.1790000000000003,"15min":4.1790000000000003},"last":0.25}]}]},{"osd":0,"up_from":9,"seq":38654707429,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314820,"kb_used_data":4936,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94057020,"statfs":{"total":96636764160,"available":96314388480,"internally_reserved":0,"allocated":5054464,"data_stored":3286977,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.26400000000000001,"5min":0.20200000000000001,"15min":0.17499999999999999},"max":{"1min":0.44500000000000001,"5min":0.46200000000000002,"15min":5.3200000000000003},"last":0.248},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.34499999999999997,"15min":0.36699999999999999},"min":{"1min":0.23899999999999999,"5min":0.222,"15min":0.17799999999999999},"max":{"1min":0.44600000000000001,"5min":0.51800000000000002,"15min":5.2039999999999997},"last":0.30299999999999999}]},{"osd":2,"last update":"Wed Apr 24 18:25:07 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34499999999999997,"15min":0.35199999999999998},"min":{"1min":0.255,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":0.46300000000000002,"5min":0.53700000000000003,"15min":4.2800000000000002},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":0.502,"5min":0.57299999999999995,"15min":7.0679999999999996},"last":0.26400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654707426,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302476,"kb_used_data":4880,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069364,"statfs":{"total":96636764160,"available":96327028736,"internally_reserved":0,"allocated":4997120,"data_stored":2696609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:38 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33300000000000002,"15min":0.35699999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":0.47199999999999998,"5min":0.55400000000000005,"15min":7.0199999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34699999999999998,"15min":0.40100000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.20699999999999999},"max":{"1min":0.752,"5min":0.752,"15min":11.084},"last":0.36499999999999999}]},{"osd":1,"last update":"Wed Apr 24 18:25:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35299999999999998,"15min":0.35999999999999999},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.216},"max":{"1min":0.46899999999999997,"5min":0.51300000000000001,"15min":5.0800000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.45000000000000001,"15min":0.38400000000000001},"min":{"1min":0.246,"5min":0.214,"15min":0.214},"max":{"1min":0.51200000000000001,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:25:49.484 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-24T18:25:49.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T18:25:49.795 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T18:25:49.797 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-24T18:25:49.808 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":6533,"stamp":"2024-04-24T18:25:49.789491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":919824,"kb_used_data":14748,"kb_used_omap":4,"kb_used_meta":904315,"kb_avail":282195696,"statfs":{"total":289910292480,"available":288968392704,"internally_reserved":0,"allocated":15101952,"data_stored":9270544,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":926018568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1042410375,"num_objects":-310,"num_object_clones":-101,"num_object_copies":-620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-310,"num_whiteouts":0,"num_read":-464786,"num_read_kb":-21481491,"num_write":-36738,"num_write_kb":-8493309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-12738579,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001907"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.956268+0000","last_change":"2024-04-24T15:59:50.776460+0000","last_active":"2024-04-24T18:25:42.956268+0000","last_peered":"2024-04-24T18:25:42.956268+0000","last_clean":"2024-04-24T18:25:42.956268+0000","last_became_active":"2024-04-24T15:59:50.776173+0000","last_became_peered":"2024-04-24T15:59:50.776173+0000","last_unstale":"2024-04-24T18:25:42.956268+0000","last_undegraded":"2024-04-24T18:25:42.956268+0000","last_fullsized":"2024-04-24T18:25:42.956268+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:54:40.948714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.579867+0000","last_change":"2024-04-24T15:59:50.774230+0000","last_active":"2024-04-24T18:25:43.579867+0000","last_peered":"2024-04-24T18:25:43.579867+0000","last_clean":"2024-04-24T18:25:43.579867+0000","last_became_active":"2024-04-24T15:59:50.773938+0000","last_became_peered":"2024-04-24T15:59:50.773938+0000","last_unstale":"2024-04-24T18:25:43.579867+0000","last_undegraded":"2024-04-24T18:25:43.579867+0000","last_fullsized":"2024-04-24T18:25:43.579867+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:54.764334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2145,"reported_epoch":2152,"state":"active+clean","last_fresh":"2024-04-24T18:25:20.143238+0000","last_change":"2024-04-24T15:59:50.775383+0000","last_active":"2024-04-24T18:25:20.143238+0000","last_peered":"2024-04-24T18:25:20.143238+0000","last_clean":"2024-04-24T18:25:20.143238+0000","last_became_active":"2024-04-24T15:59:50.775119+0000","last_became_peered":"2024-04-24T15:59:50.775119+0000","last_unstale":"2024-04-24T18:25:20.143238+0000","last_undegraded":"2024-04-24T18:25:20.143238+0000","last_fullsized":"2024-04-24T18:25:20.143238+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:20:11.243695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.953534+0000","last_change":"2024-04-24T15:59:50.775216+0000","last_active":"2024-04-24T18:25:42.953534+0000","last_peered":"2024-04-24T18:25:42.953534+0000","last_clean":"2024-04-24T18:25:42.953534+0000","last_became_active":"2024-04-24T15:59:50.774935+0000","last_became_peered":"2024-04-24T15:59:50.774935+0000","last_unstale":"2024-04-24T18:25:42.953534+0000","last_undegraded":"2024-04-24T18:25:42.953534+0000","last_fullsized":"2024-04-24T18:25:42.953534+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:50.389462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":2276,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.948563+0000","last_change":"2024-04-24T15:59:46.916688+0000","last_active":"2024-04-24T18:25:42.948563+0000","last_peered":"2024-04-24T18:25:42.948563+0000","last_clean":"2024-04-24T18:25:42.948563+0000","last_became_active":"2024-04-24T15:59:46.915637+0000","last_became_peered":"2024-04-24T15:59:46.915637+0000","last_unstale":"2024-04-24T18:25:42.948563+0000","last_undegraded":"2024-04-24T18:25:42.948563+0000","last_fullsized":"2024-04-24T18:25:42.948563+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_clean_scrub_stamp":"2024-04-24T15:59:45.750019+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:30:29.745322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2145,"reported_epoch":2152,"state":"active+clean","last_fresh":"2024-04-24T18:25:20.143006+0000","last_change":"2024-04-24T15:59:50.775501+0000","last_active":"2024-04-24T18:25:20.143006+0000","last_peered":"2024-04-24T18:25:20.143006+0000","last_clean":"2024-04-24T18:25:20.143006+0000","last_became_active":"2024-04-24T15:59:50.775157+0000","last_became_peered":"2024-04-24T15:59:50.775157+0000","last_unstale":"2024-04-24T18:25:20.143006+0000","last_undegraded":"2024-04-24T18:25:20.143006+0000","last_fullsized":"2024-04-24T18:25:20.143006+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:43:00.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.956227+0000","last_change":"2024-04-24T15:59:50.777598+0000","last_active":"2024-04-24T18:25:42.956227+0000","last_peered":"2024-04-24T18:25:42.956227+0000","last_clean":"2024-04-24T18:25:42.956227+0000","last_became_active":"2024-04-24T15:59:50.777322+0000","last_became_peered":"2024-04-24T15:59:50.777322+0000","last_unstale":"2024-04-24T18:25:42.956227+0000","last_undegraded":"2024-04-24T18:25:42.956227+0000","last_fullsized":"2024-04-24T18:25:42.956227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:54:24.332122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.962339+0000","last_change":"2024-04-24T15:59:50.774597+0000","last_active":"2024-04-24T18:25:42.962339+0000","last_peered":"2024-04-24T18:25:42.962339+0000","last_clean":"2024-04-24T18:25:42.962339+0000","last_became_active":"2024-04-24T15:59:50.774160+0000","last_became_peered":"2024-04-24T15:59:50.774160+0000","last_unstale":"2024-04-24T18:25:42.962339+0000","last_undegraded":"2024-04-24T18:25:42.962339+0000","last_fullsized":"2024-04-24T18:25:42.962339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:12.412466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.960306+0000","last_change":"2024-04-24T15:59:50.776430+0000","last_active":"2024-04-24T18:25:42.960306+0000","last_peered":"2024-04-24T18:25:42.960306+0000","last_clean":"2024-04-24T18:25:42.960306+0000","last_became_active":"2024-04-24T15:59:50.776148+0000","last_became_peered":"2024-04-24T15:59:50.776148+0000","last_unstale":"2024-04-24T18:25:42.960306+0000","last_undegraded":"2024-04-24T18:25:42.960306+0000","last_fullsized":"2024-04-24T18:25:42.960306+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:30:20.365824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949674719,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302528,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069312,"statfs":{"total":96636764160,"available":96326975488,"internally_reserved":0,"allocated":5050368,"data_stored":3286958,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"min":{"1min":0.20799999999999999,"5min":0.20100000000000001,"15min":0.17499999999999999},"max":{"1min":1.722,"5min":4.758,"15min":4.758},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.442,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":1.768,"5min":4.7919999999999998,"15min":4.7919999999999998},"last":0.26500000000000001}]},{"osd":2,"last update":"Wed Apr 24 18:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.34499999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.182},"max":{"1min":2.0680000000000001,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":0.214},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.42699999999999999,"15min":0.38700000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":0.53500000000000003,"5min":4.1790000000000003,"15min":4.1790000000000003},"last":0.25}]}]},{"osd":0,"up_from":9,"seq":38654707429,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314820,"kb_used_data":4936,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94057020,"statfs":{"total":96636764160,"available":96314388480,"internally_reserved":0,"allocated":5054464,"data_stored":3286977,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.26400000000000001,"5min":0.20200000000000001,"15min":0.17499999999999999},"max":{"1min":0.44500000000000001,"5min":0.46200000000000002,"15min":5.3200000000000003},"last":0.248},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.34499999999999997,"15min":0.36699999999999999},"min":{"1min":0.23899999999999999,"5min":0.222,"15min":0.17799999999999999},"max":{"1min":0.44600000000000001,"5min":0.51800000000000002,"15min":5.2039999999999997},"last":0.30299999999999999}]},{"osd":2,"last update":"Wed Apr 24 18:25:07 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34499999999999997,"15min":0.35199999999999998},"min":{"1min":0.255,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":0.46300000000000002,"5min":0.53700000000000003,"15min":4.2800000000000002},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":0.502,"5min":0.57299999999999995,"15min":7.0679999999999996},"last":0.26400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654707426,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302476,"kb_used_data":4880,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069364,"statfs":{"total":96636764160,"available":96327028736,"internally_reserved":0,"allocated":4997120,"data_stored":2696609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:38 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33300000000000002,"15min":0.35699999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":0.47199999999999998,"5min":0.55400000000000005,"15min":7.0199999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34699999999999998,"15min":0.40100000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.20699999999999999},"max":{"1min":0.752,"5min":0.752,"15min":11.084},"last":0.36499999999999999}]},{"osd":1,"last update":"Wed Apr 24 18:25:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35299999999999998,"15min":0.35999999999999999},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.216},"max":{"1min":0.46899999999999997,"5min":0.51300000000000001,"15min":5.0800000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.45000000000000001,"15min":0.38400000000000001},"min":{"1min":0.246,"5min":0.214,"15min":0.214},"max":{"1min":0.51200000000000001,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.38}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:25:49.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-24T18:25:50.125 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T18:25:50.125 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":2154,"fsid":"e6ccc4a4-622a-400f-a916-d56c405e9e43","created":"2024-04-24T15:59:33.767454+0000","modified":"2024-04-24T18:25:44.910351+0000","last_up_change":"2024-04-24T15:59:44.747907+0000","last_in_change":"2024-04-24T15:59:34.715818+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T15:59:45.382463+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T15:59:49.063456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cfb39152-2b32-4f85-9b4a-f2ffcc9fed42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":40663},{"type":"v1","addr":"172.21.15.136:6802","nonce":40663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":40663},{"type":"v1","addr":"172.21.15.136:6806","nonce":40663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":40663},{"type":"v1","addr":"172.21.15.136:6814","nonce":40663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":40663},{"type":"v1","addr":"172.21.15.136:6810","nonce":40663}]},"public_addr":"172.21.15.136:6802/40663","cluster_addr":"172.21.15.136:6806/40663","heartbeat_back_addr":"172.21.15.136:6814/40663","heartbeat_front_addr":"172.21.15.136:6810/40663","state":["exists","up"]},{"osd":1,"uuid":"80d9ea77-6b87-44c8-bf04-a95b22ad41f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":40665},{"type":"v1","addr":"172.21.15.136:6817","nonce":40665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":40665},{"type":"v1","addr":"172.21.15.136:6819","nonce":40665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":40665},{"type":"v1","addr":"172.21.15.136:6823","nonce":40665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":40665},{"type":"v1","addr":"172.21.15.136:6821","nonce":40665}]},"public_addr":"172.21.15.136:6817/40665","cluster_addr":"172.21.15.136:6819/40665","heartbeat_back_addr":"172.21.15.136:6823/40665","heartbeat_front_addr":"172.21.15.136:6821/40665","state":["exists","up"]},{"osd":2,"uuid":"240de483-dd6c-4282-ad4c-f5fdb8a32545","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":40664},{"type":"v1","addr":"172.21.15.136:6803","nonce":40664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6805","nonce":40664},{"type":"v1","addr":"172.21.15.136:6807","nonce":40664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6813","nonce":40664},{"type":"v1","addr":"172.21.15.136:6815","nonce":40664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6809","nonce":40664},{"type":"v1","addr":"172.21.15.136:6811","nonce":40664}]},"public_addr":"172.21.15.136:6803/40664","cluster_addr":"172.21.15.136:6807/40664","heartbeat_back_addr":"172.21.15.136:6815/40664","heartbeat_front_addr":"172.21.15.136:6811/40664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.824415+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.805861+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:39.799092+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T18:25:51.140 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-04-24T18:25:51.140 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T18:25:51.386 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-24T18:25:51.386 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T18:25:51.386 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-24T18:25:51.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-04-24T18:25:51.715 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T18:25:51.726 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-04-24T18:25:51.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T18:25:51.868 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-24T18:25:51.868 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T18:25:51.868 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-24T18:25:51.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-04-24T18:25:52.195 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T18:25:52.206 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-04-24T18:25:52.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T18:25:52.347 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-24T18:25:52.348 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T18:25:52.348 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-24T18:25:52.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-04-24T18:25:52.676 INFO:teuthology.orchestra.run.smithi136.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T18:25:52.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T18:25:52.999 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T18:25:53.002 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-24T18:25:53.013 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":6534,"stamp":"2024-04-24T18:25:51.789787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":919132,"kb_used_data":14056,"kb_used_omap":4,"kb_used_meta":904315,"kb_avail":282196388,"statfs":{"total":289910292480,"available":288969101312,"internally_reserved":0,"allocated":14393344,"data_stored":8884154,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":926018568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":-4652,"num_read_kb":-4646,"num_write":-297,"num_write_kb":-295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-19,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.882264"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.956268+0000","last_change":"2024-04-24T15:59:50.776460+0000","last_active":"2024-04-24T18:25:42.956268+0000","last_peered":"2024-04-24T18:25:42.956268+0000","last_clean":"2024-04-24T18:25:42.956268+0000","last_became_active":"2024-04-24T15:59:50.776173+0000","last_became_peered":"2024-04-24T15:59:50.776173+0000","last_unstale":"2024-04-24T18:25:42.956268+0000","last_undegraded":"2024-04-24T18:25:42.956268+0000","last_fullsized":"2024-04-24T18:25:42.956268+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:54:40.948714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.579867+0000","last_change":"2024-04-24T15:59:50.774230+0000","last_active":"2024-04-24T18:25:43.579867+0000","last_peered":"2024-04-24T18:25:43.579867+0000","last_clean":"2024-04-24T18:25:43.579867+0000","last_became_active":"2024-04-24T15:59:50.773938+0000","last_became_peered":"2024-04-24T15:59:50.773938+0000","last_unstale":"2024-04-24T18:25:43.579867+0000","last_undegraded":"2024-04-24T18:25:43.579867+0000","last_fullsized":"2024-04-24T18:25:43.579867+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:34:54.764334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.933460+0000","last_change":"2024-04-24T15:59:50.775383+0000","last_active":"2024-04-24T18:25:42.933460+0000","last_peered":"2024-04-24T18:25:42.933460+0000","last_clean":"2024-04-24T18:25:42.933460+0000","last_became_active":"2024-04-24T15:59:50.775119+0000","last_became_peered":"2024-04-24T15:59:50.775119+0000","last_unstale":"2024-04-24T18:25:42.933460+0000","last_undegraded":"2024-04-24T18:25:42.933460+0000","last_fullsized":"2024-04-24T18:25:42.933460+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:20:11.243695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.953534+0000","last_change":"2024-04-24T15:59:50.775216+0000","last_active":"2024-04-24T18:25:42.953534+0000","last_peered":"2024-04-24T18:25:42.953534+0000","last_clean":"2024-04-24T18:25:42.953534+0000","last_became_active":"2024-04-24T15:59:50.774935+0000","last_became_peered":"2024-04-24T15:59:50.774935+0000","last_unstale":"2024-04-24T18:25:42.953534+0000","last_undegraded":"2024-04-24T18:25:42.953534+0000","last_fullsized":"2024-04-24T18:25:42.953534+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:50:50.389462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":2276,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.948563+0000","last_change":"2024-04-24T15:59:46.916688+0000","last_active":"2024-04-24T18:25:42.948563+0000","last_peered":"2024-04-24T18:25:42.948563+0000","last_clean":"2024-04-24T18:25:42.948563+0000","last_became_active":"2024-04-24T15:59:46.915637+0000","last_became_peered":"2024-04-24T15:59:46.915637+0000","last_unstale":"2024-04-24T18:25:42.948563+0000","last_undegraded":"2024-04-24T18:25:42.948563+0000","last_fullsized":"2024-04-24T18:25:42.948563+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:45.750019+0000","last_clean_scrub_stamp":"2024-04-24T15:59:45.750019+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:30:29.745322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.931788+0000","last_change":"2024-04-24T15:59:50.775501+0000","last_active":"2024-04-24T18:25:42.931788+0000","last_peered":"2024-04-24T18:25:42.931788+0000","last_clean":"2024-04-24T18:25:42.931788+0000","last_became_active":"2024-04-24T15:59:50.775157+0000","last_became_peered":"2024-04-24T15:59:50.775157+0000","last_unstale":"2024-04-24T18:25:42.931788+0000","last_undegraded":"2024-04-24T18:25:42.931788+0000","last_fullsized":"2024-04-24T18:25:42.931788+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:43:00.469662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.956227+0000","last_change":"2024-04-24T15:59:50.777598+0000","last_active":"2024-04-24T18:25:42.956227+0000","last_peered":"2024-04-24T18:25:42.956227+0000","last_clean":"2024-04-24T18:25:42.956227+0000","last_became_active":"2024-04-24T15:59:50.777322+0000","last_became_peered":"2024-04-24T15:59:50.777322+0000","last_unstale":"2024-04-24T18:25:42.956227+0000","last_undegraded":"2024-04-24T18:25:42.956227+0000","last_fullsized":"2024-04-24T18:25:42.956227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:54:24.332122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.962339+0000","last_change":"2024-04-24T15:59:50.774597+0000","last_active":"2024-04-24T18:25:42.962339+0000","last_peered":"2024-04-24T18:25:42.962339+0000","last_clean":"2024-04-24T18:25:42.962339+0000","last_became_active":"2024-04-24T15:59:50.774160+0000","last_became_peered":"2024-04-24T15:59:50.774160+0000","last_unstale":"2024-04-24T18:25:42.962339+0000","last_undegraded":"2024-04-24T18:25:42.962339+0000","last_fullsized":"2024-04-24T18:25:42.962339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:12.412466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2146,"reported_epoch":2153,"state":"active+clean","last_fresh":"2024-04-24T18:25:42.960306+0000","last_change":"2024-04-24T15:59:50.776430+0000","last_active":"2024-04-24T18:25:42.960306+0000","last_peered":"2024-04-24T18:25:42.960306+0000","last_clean":"2024-04-24T18:25:42.960306+0000","last_became_active":"2024-04-24T15:59:50.776148+0000","last_became_peered":"2024-04-24T15:59:50.776148+0000","last_unstale":"2024-04-24T18:25:42.960306+0000","last_undegraded":"2024-04-24T18:25:42.960306+0000","last_fullsized":"2024-04-24T18:25:42.960306+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:30:20.365824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949674720,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302300,"kb_used_data":4704,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069540,"statfs":{"total":96636764160,"available":96327208960,"internally_reserved":0,"allocated":4816896,"data_stored":3158174,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"min":{"1min":0.20799999999999999,"5min":0.20100000000000001,"15min":0.17499999999999999},"max":{"1min":1.722,"5min":4.758,"15min":4.758},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.442,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":1.768,"5min":4.7919999999999998,"15min":4.7919999999999998},"last":0.54800000000000004}]},{"osd":2,"last update":"Wed Apr 24 18:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.34499999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.182},"max":{"1min":2.0680000000000001,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":0.44},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.42699999999999999,"15min":0.38700000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":0.53500000000000003,"5min":4.1790000000000003,"15min":4.1790000000000003},"last":0.48699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654707430,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314588,"kb_used_data":4704,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94057252,"statfs":{"total":96636764160,"available":96314626048,"internally_reserved":0,"allocated":4816896,"data_stored":3158174,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:25:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.33700000000000002,"15min":0.33900000000000002},"min":{"1min":0.26400000000000001,"5min":0.20200000000000001,"15min":0.17499999999999999},"max":{"1min":0.44500000000000001,"5min":0.46200000000000002,"15min":5.3200000000000003},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.34499999999999997,"15min":0.36699999999999999},"min":{"1min":0.23899999999999999,"5min":0.222,"15min":0.17799999999999999},"max":{"1min":0.44600000000000001,"5min":0.51800000000000002,"15min":5.2039999999999997},"last":0.27100000000000002}]},{"osd":2,"last update":"Wed Apr 24 18:25:07 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34499999999999997,"15min":0.35199999999999998},"min":{"1min":0.255,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":0.46300000000000002,"5min":0.53700000000000003,"15min":4.2800000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39100000000000001,"15min":0.42899999999999999},"min":{"1min":0.30199999999999999,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":0.502,"5min":0.57299999999999995,"15min":7.0679999999999996},"last":0.29799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654707427,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302244,"kb_used_data":4648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069596,"statfs":{"total":96636764160,"available":96327266304,"internally_reserved":0,"allocated":4759552,"data_stored":2567806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:38 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33300000000000002,"15min":0.35699999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":0.47199999999999998,"5min":0.55400000000000005,"15min":7.0199999999999996},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34699999999999998,"15min":0.40100000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.20699999999999999},"max":{"1min":0.752,"5min":0.752,"15min":11.084},"last":0.433}]},{"osd":1,"last update":"Wed Apr 24 18:25:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35299999999999998,"15min":0.35999999999999999},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.216},"max":{"1min":0.46899999999999997,"5min":0.51300000000000001,"15min":5.0800000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.45000000000000001,"15min":0.38400000000000001},"min":{"1min":0.246,"5min":0.214,"15min":0.214},"max":{"1min":0.51200000000000001,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.36399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:25:53.016 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-24T15:59:49.759412+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=25, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:25:53.016 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-24T15:59:49.759412+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=25, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:25:53.016 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-24T15:59:49.759412+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=25, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:25:53.017 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-24T15:59:49.759412+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=25, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:25:53.017 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T15:59:45.750019+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=45, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=25, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:25:53.017 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T15:59:49.759412+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=25, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:25:53.017 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-24T15:59:49.759412+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=25, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:25:53.017 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T15:59:49.759412+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=25, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:25:53.017 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-24T15:59:49.759412+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=25, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:25:53.017 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T18:26:13.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T18:26:13.343 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T18:26:13.344 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-24T18:26:13.356 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":6544,"stamp":"2024-04-24T18:26:11.793692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":918400,"kb_used_data":14056,"kb_used_omap":4,"kb_used_meta":904315,"kb_avail":282197120,"statfs":{"total":289910292480,"available":288969850880,"internally_reserved":0,"allocated":14393344,"data_stored":8884154,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":926018568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002271"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:54.399694+0000","last_change":"2024-04-24T18:25:54.399654+0000","last_active":"2024-04-24T18:25:54.399694+0000","last_peered":"2024-04-24T18:25:54.399694+0000","last_clean":"2024-04-24T18:25:54.399694+0000","last_became_active":"2024-04-24T15:59:50.776173+0000","last_became_peered":"2024-04-24T15:59:50.776173+0000","last_unstale":"2024-04-24T18:25:54.399694+0000","last_undegraded":"2024-04-24T18:25:54.399694+0000","last_fullsized":"2024-04-24T18:25:54.399694+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:25:54.399521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:25:54.399521+0000","last_clean_scrub_stamp":"2024-04-24T18:25:54.399521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:06.104119+0000","scrub_duration":0.0016290720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:57.500079+0000","last_change":"2024-04-24T18:25:57.500050+0000","last_active":"2024-04-24T18:25:57.500079+0000","last_peered":"2024-04-24T18:25:57.500079+0000","last_clean":"2024-04-24T18:25:57.500079+0000","last_became_active":"2024-04-24T15:59:50.773938+0000","last_became_peered":"2024-04-24T15:59:50.773938+0000","last_unstale":"2024-04-24T18:25:57.500079+0000","last_undegraded":"2024-04-24T18:25:57.500079+0000","last_fullsized":"2024-04-24T18:25:57.500079+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:25:57.499948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:25:57.499948+0000","last_clean_scrub_stamp":"2024-04-24T18:25:57.499948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:38:33.437836+0000","scrub_duration":0.0071885539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:55.316069+0000","last_change":"2024-04-24T18:25:55.316034+0000","last_active":"2024-04-24T18:25:55.316069+0000","last_peered":"2024-04-24T18:25:55.316069+0000","last_clean":"2024-04-24T18:25:55.316069+0000","last_became_active":"2024-04-24T15:59:50.775119+0000","last_became_peered":"2024-04-24T15:59:50.775119+0000","last_unstale":"2024-04-24T18:25:55.316069+0000","last_undegraded":"2024-04-24T18:25:55.316069+0000","last_fullsized":"2024-04-24T18:25:55.316069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:25:55.315919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:25:55.315919+0000","last_clean_scrub_stamp":"2024-04-24T18:25:55.315919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:53.798561+0000","scrub_duration":0.019960820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:26:06.279105+0000","last_change":"2024-04-24T18:26:06.279062+0000","last_active":"2024-04-24T18:26:06.279105+0000","last_peered":"2024-04-24T18:26:06.279105+0000","last_clean":"2024-04-24T18:26:06.279105+0000","last_became_active":"2024-04-24T15:59:50.774935+0000","last_became_peered":"2024-04-24T15:59:50.774935+0000","last_unstale":"2024-04-24T18:26:06.279105+0000","last_undegraded":"2024-04-24T18:26:06.279105+0000","last_fullsized":"2024-04-24T18:26:06.279105+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:06.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:06.278910+0000","last_clean_scrub_stamp":"2024-04-24T18:26:06.278910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:04.448840+0000","scrub_duration":0.028426283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"2154'99","reported_seq":2289,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:52.423859+0000","last_change":"2024-04-24T18:25:52.423817+0000","last_active":"2024-04-24T18:25:52.423859+0000","last_peered":"2024-04-24T18:25:52.423859+0000","last_clean":"2024-04-24T18:25:52.423859+0000","last_became_active":"2024-04-24T15:59:46.915637+0000","last_became_peered":"2024-04-24T15:59:46.915637+0000","last_unstale":"2024-04-24T18:25:52.423859+0000","last_undegraded":"2024-04-24T18:25:52.423859+0000","last_fullsized":"2024-04-24T18:25:52.423859+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"2154'99","last_scrub_stamp":"2024-04-24T18:25:52.423695+0000","last_deep_scrub":"2154'99","last_deep_scrub_stamp":"2024-04-24T18:25:52.423695+0000","last_clean_scrub_stamp":"2024-04-24T18:25:52.423695+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:13:25.044273+0000","scrub_duration":0.0097165169999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:26:00.320620+0000","last_change":"2024-04-24T18:26:00.320595+0000","last_active":"2024-04-24T18:26:00.320620+0000","last_peered":"2024-04-24T18:26:00.320620+0000","last_clean":"2024-04-24T18:26:00.320620+0000","last_became_active":"2024-04-24T15:59:50.775157+0000","last_became_peered":"2024-04-24T15:59:50.775157+0000","last_unstale":"2024-04-24T18:26:00.320620+0000","last_undegraded":"2024-04-24T18:26:00.320620+0000","last_fullsized":"2024-04-24T18:26:00.320620+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:00.320487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:00.320487+0000","last_clean_scrub_stamp":"2024-04-24T18:26:00.320487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:43:47.322358+0000","scrub_duration":0.010694874,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:26:00.340821+0000","last_change":"2024-04-24T18:26:00.340757+0000","last_active":"2024-04-24T18:26:00.340821+0000","last_peered":"2024-04-24T18:26:00.340821+0000","last_clean":"2024-04-24T18:26:00.340821+0000","last_became_active":"2024-04-24T15:59:50.777322+0000","last_became_peered":"2024-04-24T15:59:50.777322+0000","last_unstale":"2024-04-24T18:26:00.340821+0000","last_undegraded":"2024-04-24T18:26:00.340821+0000","last_fullsized":"2024-04-24T18:26:00.340821+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:00.340595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:00.340595+0000","last_clean_scrub_stamp":"2024-04-24T18:26:00.340595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:55.048421+0000","scrub_duration":0.019603252000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:44.920372+0000","last_change":"2024-04-24T15:59:50.774597+0000","last_active":"2024-04-24T18:25:44.920372+0000","last_peered":"2024-04-24T18:25:44.920372+0000","last_clean":"2024-04-24T18:25:44.920372+0000","last_became_active":"2024-04-24T15:59:50.774160+0000","last_became_peered":"2024-04-24T15:59:50.774160+0000","last_unstale":"2024-04-24T18:25:44.920372+0000","last_undegraded":"2024-04-24T18:25:44.920372+0000","last_fullsized":"2024-04-24T18:25:44.920372+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:49.759412+0000","last_clean_scrub_stamp":"2024-04-24T15:59:49.759412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:26:04.296190+0000","last_change":"2024-04-24T18:26:04.296144+0000","last_active":"2024-04-24T18:26:04.296190+0000","last_peered":"2024-04-24T18:26:04.296190+0000","last_clean":"2024-04-24T18:26:04.296190+0000","last_became_active":"2024-04-24T15:59:50.776148+0000","last_became_peered":"2024-04-24T15:59:50.776148+0000","last_unstale":"2024-04-24T18:26:04.296190+0000","last_undegraded":"2024-04-24T18:26:04.296190+0000","last_fullsized":"2024-04-24T18:26:04.296190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:04.296017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:04.296017+0000","last_clean_scrub_stamp":"2024-04-24T18:26:04.296017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:56:50.866570+0000","scrub_duration":0.031947494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949674724,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":4704,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":4816896,"data_stored":3158174,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.375,"15min":0.33300000000000002},"min":{"1min":0.20799999999999999,"5min":0.20100000000000001,"15min":0.17499999999999999},"max":{"1min":1.722,"5min":4.758,"15min":4.758},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.442,"15min":0.40000000000000002},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.23899999999999999},"max":{"1min":1.768,"5min":4.7919999999999998,"15min":4.7919999999999998},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Apr 24 18:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.38900000000000001,"15min":0.34499999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.182},"max":{"1min":2.0680000000000001,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.42699999999999999,"15min":0.38700000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20899999999999999},"max":{"1min":0.53500000000000003,"5min":4.1790000000000003,"15min":4.1790000000000003},"last":0.31}]}]},{"osd":0,"up_from":9,"seq":38654707434,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314344,"kb_used_data":4704,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94057496,"statfs":{"total":96636764160,"available":96314875904,"internally_reserved":0,"allocated":4816896,"data_stored":3158174,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:26:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.35799999999999998,"15min":0.34599999999999997},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.17499999999999999},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":5.3200000000000003},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.307,"5min":0.34200000000000003,"15min":0.36499999999999999},"min":{"1min":0.248,"5min":0.222,"15min":0.19600000000000001},"max":{"1min":0.44900000000000001,"5min":0.51800000000000002,"15min":5.2039999999999997},"last":0.308}]},{"osd":2,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34799999999999998,"15min":0.34999999999999998},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":0.52500000000000002,"5min":0.53700000000000003,"15min":4.2800000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.40200000000000002,"15min":0.432},"min":{"1min":0.26400000000000001,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":7.0679999999999996},"last":0.373}]}]},{"osd":2,"up_from":9,"seq":38654707431,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302000,"kb_used_data":4648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069840,"statfs":{"total":96636764160,"available":96327516160,"internally_reserved":0,"allocated":4759552,"data_stored":2567806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:38 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33300000000000002,"15min":0.35699999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":0.47199999999999998,"5min":0.55400000000000005,"15min":7.0199999999999996},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34699999999999998,"15min":0.40100000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.20699999999999999},"max":{"1min":0.752,"5min":0.752,"15min":11.084},"last":0.378}]},{"osd":1,"last update":"Wed Apr 24 18:25:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35299999999999998,"15min":0.35999999999999999},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.216},"max":{"1min":0.46899999999999997,"5min":0.51300000000000001,"15min":5.0800000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.45000000000000001,"15min":0.38400000000000001},"min":{"1min":0.246,"5min":0.214,"15min":0.214},"max":{"1min":0.51200000000000001,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.34300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:26:13.357 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T15:59:49.759412+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=25, tm_sec=50, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:26:13.357 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T18:26:33.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-24T18:26:33.684 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-24T18:26:33.686 INFO:teuthology.orchestra.run.smithi136.stderr:dumped all 2024-04-24T18:26:33.697 INFO:teuthology.orchestra.run.smithi136.stdout:{"pg_ready":true,"pg_map":{"version":6554,"stamp":"2024-04-24T18:26:31.797822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":918400,"kb_used_data":14056,"kb_used_omap":4,"kb_used_meta":904315,"kb_avail":282197120,"statfs":{"total":289910292480,"available":288969850880,"internally_reserved":0,"allocated":14393344,"data_stored":8884154,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":926018568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002682"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:54.399694+0000","last_change":"2024-04-24T18:25:54.399654+0000","last_active":"2024-04-24T18:25:54.399694+0000","last_peered":"2024-04-24T18:25:54.399694+0000","last_clean":"2024-04-24T18:25:54.399694+0000","last_became_active":"2024-04-24T15:59:50.776173+0000","last_became_peered":"2024-04-24T15:59:50.776173+0000","last_unstale":"2024-04-24T18:25:54.399694+0000","last_undegraded":"2024-04-24T18:25:54.399694+0000","last_fullsized":"2024-04-24T18:25:54.399694+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:25:54.399521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:25:54.399521+0000","last_clean_scrub_stamp":"2024-04-24T18:25:54.399521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:06.104119+0000","scrub_duration":0.0016290720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:57.500079+0000","last_change":"2024-04-24T18:25:57.500050+0000","last_active":"2024-04-24T18:25:57.500079+0000","last_peered":"2024-04-24T18:25:57.500079+0000","last_clean":"2024-04-24T18:25:57.500079+0000","last_became_active":"2024-04-24T15:59:50.773938+0000","last_became_peered":"2024-04-24T15:59:50.773938+0000","last_unstale":"2024-04-24T18:25:57.500079+0000","last_undegraded":"2024-04-24T18:25:57.500079+0000","last_fullsized":"2024-04-24T18:25:57.500079+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:25:57.499948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:25:57.499948+0000","last_clean_scrub_stamp":"2024-04-24T18:25:57.499948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:38:33.437836+0000","scrub_duration":0.0071885539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:55.316069+0000","last_change":"2024-04-24T18:25:55.316034+0000","last_active":"2024-04-24T18:25:55.316069+0000","last_peered":"2024-04-24T18:25:55.316069+0000","last_clean":"2024-04-24T18:25:55.316069+0000","last_became_active":"2024-04-24T15:59:50.775119+0000","last_became_peered":"2024-04-24T15:59:50.775119+0000","last_unstale":"2024-04-24T18:25:55.316069+0000","last_undegraded":"2024-04-24T18:25:55.316069+0000","last_fullsized":"2024-04-24T18:25:55.316069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:25:55.315919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:25:55.315919+0000","last_clean_scrub_stamp":"2024-04-24T18:25:55.315919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:17:53.798561+0000","scrub_duration":0.019960820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:26:06.279105+0000","last_change":"2024-04-24T18:26:06.279062+0000","last_active":"2024-04-24T18:26:06.279105+0000","last_peered":"2024-04-24T18:26:06.279105+0000","last_clean":"2024-04-24T18:26:06.279105+0000","last_became_active":"2024-04-24T15:59:50.774935+0000","last_became_peered":"2024-04-24T15:59:50.774935+0000","last_unstale":"2024-04-24T18:26:06.279105+0000","last_undegraded":"2024-04-24T18:26:06.279105+0000","last_fullsized":"2024-04-24T18:26:06.279105+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:06.278910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:06.278910+0000","last_clean_scrub_stamp":"2024-04-24T18:26:06.278910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:55:04.448840+0000","scrub_duration":0.028426283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"2154'99","reported_seq":2289,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:52.423859+0000","last_change":"2024-04-24T18:25:52.423817+0000","last_active":"2024-04-24T18:25:52.423859+0000","last_peered":"2024-04-24T18:25:52.423859+0000","last_clean":"2024-04-24T18:25:52.423859+0000","last_became_active":"2024-04-24T15:59:46.915637+0000","last_became_peered":"2024-04-24T15:59:46.915637+0000","last_unstale":"2024-04-24T18:25:52.423859+0000","last_undegraded":"2024-04-24T18:25:52.423859+0000","last_fullsized":"2024-04-24T18:25:52.423859+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"2154'99","last_scrub_stamp":"2024-04-24T18:25:52.423695+0000","last_deep_scrub":"2154'99","last_deep_scrub_stamp":"2024-04-24T18:25:52.423695+0000","last_clean_scrub_stamp":"2024-04-24T18:25:52.423695+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:13:25.044273+0000","scrub_duration":0.0097165169999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:26:00.320620+0000","last_change":"2024-04-24T18:26:00.320595+0000","last_active":"2024-04-24T18:26:00.320620+0000","last_peered":"2024-04-24T18:26:00.320620+0000","last_clean":"2024-04-24T18:26:00.320620+0000","last_became_active":"2024-04-24T15:59:50.775157+0000","last_became_peered":"2024-04-24T15:59:50.775157+0000","last_unstale":"2024-04-24T18:26:00.320620+0000","last_undegraded":"2024-04-24T18:26:00.320620+0000","last_fullsized":"2024-04-24T18:26:00.320620+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:00.320487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:00.320487+0000","last_clean_scrub_stamp":"2024-04-24T18:26:00.320487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:43:47.322358+0000","scrub_duration":0.010694874,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:26:00.340821+0000","last_change":"2024-04-24T18:26:00.340757+0000","last_active":"2024-04-24T18:26:00.340821+0000","last_peered":"2024-04-24T18:26:00.340821+0000","last_clean":"2024-04-24T18:26:00.340821+0000","last_became_active":"2024-04-24T15:59:50.777322+0000","last_became_peered":"2024-04-24T15:59:50.777322+0000","last_unstale":"2024-04-24T18:26:00.340821+0000","last_undegraded":"2024-04-24T18:26:00.340821+0000","last_fullsized":"2024-04-24T18:26:00.340821+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:00.340595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:00.340595+0000","last_clean_scrub_stamp":"2024-04-24T18:26:00.340595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:42:55.048421+0000","scrub_duration":0.019603252000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:26:17.234424+0000","last_change":"2024-04-24T18:26:17.234382+0000","last_active":"2024-04-24T18:26:17.234424+0000","last_peered":"2024-04-24T18:26:17.234424+0000","last_clean":"2024-04-24T18:26:17.234424+0000","last_became_active":"2024-04-24T15:59:50.774160+0000","last_became_peered":"2024-04-24T15:59:50.774160+0000","last_unstale":"2024-04-24T18:26:17.234424+0000","last_undegraded":"2024-04-24T18:26:17.234424+0000","last_fullsized":"2024-04-24T18:26:17.234424+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:17.234235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:17.234235+0000","last_clean_scrub_stamp":"2024-04-24T18:26:17.234235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:36:13.905049+0000","scrub_duration":0.0081557820000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2157,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:26:04.296190+0000","last_change":"2024-04-24T18:26:04.296144+0000","last_active":"2024-04-24T18:26:04.296190+0000","last_peered":"2024-04-24T18:26:04.296190+0000","last_clean":"2024-04-24T18:26:04.296190+0000","last_became_active":"2024-04-24T15:59:50.776148+0000","last_became_peered":"2024-04-24T15:59:50.776148+0000","last_unstale":"2024-04-24T18:26:04.296190+0000","last_undegraded":"2024-04-24T18:26:04.296190+0000","last_fullsized":"2024-04-24T18:26:04.296190+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:04.296017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:04.296017+0000","last_clean_scrub_stamp":"2024-04-24T18:26:04.296017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:56:50.866570+0000","scrub_duration":0.031947494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6816,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949674728,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":4704,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":4816896,"data_stored":3158174,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:26:23 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.32600000000000001,"15min":0.33300000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.17499999999999999},"max":{"1min":0.51200000000000001,"5min":1.722,"15min":4.758},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.39300000000000002,"15min":0.40000000000000002},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23899999999999999},"max":{"1min":0.54800000000000004,"5min":1.768,"15min":4.7919999999999998},"last":0.53600000000000003}]},{"osd":2,"last update":"Wed Apr 24 18:26:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.38800000000000001,"15min":0.35099999999999998},"min":{"1min":0.214,"5min":0.20999999999999999,"15min":0.182},"max":{"1min":1.756,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.38300000000000001,"15min":0.38500000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.20899999999999999},"max":{"1min":0.64200000000000002,"5min":2.6890000000000001,"15min":4.1790000000000003},"last":0.51900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654707438,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314344,"kb_used_data":4704,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94057496,"statfs":{"total":96636764160,"available":96314875904,"internally_reserved":0,"allocated":4816896,"data_stored":3158174,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:26:05 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.35799999999999998,"15min":0.34599999999999997},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.17499999999999999},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":5.3200000000000003},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.307,"5min":0.34200000000000003,"15min":0.36499999999999999},"min":{"1min":0.248,"5min":0.222,"15min":0.19600000000000001},"max":{"1min":0.44900000000000001,"5min":0.51800000000000002,"15min":5.2039999999999997},"last":0.32200000000000001}]},{"osd":2,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34799999999999998,"15min":0.34999999999999998},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.191},"max":{"1min":0.52500000000000002,"5min":0.53700000000000003,"15min":4.2800000000000002},"last":0.31},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.40200000000000002,"15min":0.432},"min":{"1min":0.26400000000000001,"5min":0.23200000000000001,"15min":0.221},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":7.0679999999999996},"last":0.34399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654707435,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302000,"kb_used_data":4648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94069840,"statfs":{"total":96636764160,"available":96327516160,"internally_reserved":0,"allocated":4759552,"data_stored":2567806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:38 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33300000000000002,"15min":0.35699999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.17599999999999999},"max":{"1min":0.47199999999999998,"5min":0.55400000000000005,"15min":7.0199999999999996},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.34699999999999998,"15min":0.40100000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.20699999999999999},"max":{"1min":0.752,"5min":0.752,"15min":11.084},"last":0.375}]},{"osd":1,"last update":"Wed Apr 24 18:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35099999999999998,"15min":0.36099999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":5.0800000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.36299999999999999,"15min":0.38400000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.214},"max":{"1min":0.47099999999999997,"5min":0.51200000000000001,"15min":7.0540000000000003},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3408,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:26:33.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-04-24T18:26:34.029 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T18:26:34.029 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T18:26:34.030 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-04-24T18:26:34.030 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T18:26:34.083 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-04-24T18:26:34.084 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-04-24T18:26:34.084 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T18:26:34.144 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-04-24T18:26:34.144 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-04-24T18:26:34.145 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T18:26:34.197 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-04-24T18:26:34.197 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T18:26:34.198 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-04-24T18:26:34.198 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T18:26:34.227 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-04-24T18:26:34.227 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T18:26:34.228 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-04-24T18:26:34.228 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T18:26:34.245 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-04-24T18:26:34.245 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T18:26:34.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-24T18:26:34.501 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T18:26:34.501 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-04-24T18:26:34.670 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T18:26:34.670 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-04-24T18:26:34.801 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T18:26:34.802 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-04-24T18:26:34.878 INFO:tasks.ceph:Archiving mon data... 2024-04-24T18:26:34.879 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592/data/cluster2.mon.a.tgz 2024-04-24T18:26:34.880 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-04-24T18:26:34.897 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.jEkgbodmR8 2024-04-24T18:26:34.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.jEkgbodmR8 2024-04-24T18:26:35.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.jEkgbodmR8 2024-04-24T18:26:35.193 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.jEkgbodmR8 is 592KB 2024-04-24T18:26:35.267 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.jEkgbodmR8 2024-04-24T18:26:35.283 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T18:26:35.284 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T18:26:35.286 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T18:26:35.365 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T18:26:35.374 INFO:tasks.ceph.ceph_manager.cluster1:Canceling any pending splits or merges... 2024-04-24T18:26:35.375 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T18:26:35.733 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T18:26:35.734 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":2155,"fsid":"b9870ca7-a3e3-48c6-875b-287e67643414","created":"2024-04-24T15:58:55.135792+0000","modified":"2024-04-24T18:25:45.507181+0000","last_up_change":"2024-04-24T15:59:06.732126+0000","last_in_change":"2024-04-24T15:58:56.181720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T15:59:07.567611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T15:59:10.739509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"6fc36614-9a61-4a2a-889a-bf72224374da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":40015},{"type":"v1","addr":"172.21.15.42:6809","nonce":40015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":40015},{"type":"v1","addr":"172.21.15.42:6811","nonce":40015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":40015},{"type":"v1","addr":"172.21.15.42:6815","nonce":40015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":40015},{"type":"v1","addr":"172.21.15.42:6813","nonce":40015}]},"public_addr":"172.21.15.42:6809/40015","cluster_addr":"172.21.15.42:6811/40015","heartbeat_back_addr":"172.21.15.42:6815/40015","heartbeat_front_addr":"172.21.15.42:6813/40015","state":["exists","up"]},{"osd":1,"uuid":"bca44732-0beb-4819-9928-ca629336d3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":40016},{"type":"v1","addr":"172.21.15.42:6817","nonce":40016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":40016},{"type":"v1","addr":"172.21.15.42:6819","nonce":40016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":40016},{"type":"v1","addr":"172.21.15.42:6823","nonce":40016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":40016},{"type":"v1","addr":"172.21.15.42:6821","nonce":40016}]},"public_addr":"172.21.15.42:6817/40016","cluster_addr":"172.21.15.42:6819/40016","heartbeat_back_addr":"172.21.15.42:6823/40016","heartbeat_front_addr":"172.21.15.42:6821/40016","state":["exists","up"]},{"osd":2,"uuid":"56f0f362-3f29-4db1-8086-a02243c2e017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":40000},{"type":"v1","addr":"172.21.15.42:6801","nonce":40000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":40000},{"type":"v1","addr":"172.21.15.42:6803","nonce":40000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":40000},{"type":"v1","addr":"172.21.15.42:6807","nonce":40000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":40000},{"type":"v1","addr":"172.21.15.42:6805","nonce":40000}]},"public_addr":"172.21.15.42:6801/40000","cluster_addr":"172.21.15.42:6803/40000","heartbeat_back_addr":"172.21.15.42:6807/40000","heartbeat_front_addr":"172.21.15.42:6805/40000","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.362853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.366407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.313848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T18:26:35.750 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-24T18:26:35.750 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T18:26:36.101 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T18:26:36.104 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-04-24T18:26:36.117 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":6575,"stamp":"2024-04-24T18:26:35.892119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":931736,"kb_used_data":14128,"kb_used_omap":4,"kb_used_meta":916603,"kb_avail":282183784,"statfs":{"total":289910292480,"available":288956194816,"internally_reserved":0,"allocated":14467072,"data_stored":8892698,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":938601480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001916"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.665727+0000","last_change":"2024-04-24T15:59:12.355878+0000","last_active":"2024-04-24T18:25:43.665727+0000","last_peered":"2024-04-24T18:25:43.665727+0000","last_clean":"2024-04-24T18:25:43.665727+0000","last_became_active":"2024-04-24T15:59:12.355473+0000","last_became_peered":"2024-04-24T15:59:12.355473+0000","last_unstale":"2024-04-24T18:25:43.665727+0000","last_undegraded":"2024-04-24T18:25:43.665727+0000","last_fullsized":"2024-04-24T18:25:43.665727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:37:23.943905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.717028+0000","last_change":"2024-04-24T15:59:12.356273+0000","last_active":"2024-04-24T18:25:43.717028+0000","last_peered":"2024-04-24T18:25:43.717028+0000","last_clean":"2024-04-24T18:25:43.717028+0000","last_became_active":"2024-04-24T15:59:12.355976+0000","last_became_peered":"2024-04-24T15:59:12.355976+0000","last_unstale":"2024-04-24T18:25:43.717028+0000","last_undegraded":"2024-04-24T18:25:43.717028+0000","last_fullsized":"2024-04-24T18:25:43.717028+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:14:20.180867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.509224+0000","last_change":"2024-04-24T15:59:12.357100+0000","last_active":"2024-04-24T18:25:43.509224+0000","last_peered":"2024-04-24T18:25:43.509224+0000","last_clean":"2024-04-24T18:25:43.509224+0000","last_became_active":"2024-04-24T15:59:12.356512+0000","last_became_peered":"2024-04-24T15:59:12.356512+0000","last_unstale":"2024-04-24T18:25:43.509224+0000","last_undegraded":"2024-04-24T18:25:43.509224+0000","last_fullsized":"2024-04-24T18:25:43.509224+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:40.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.662385+0000","last_change":"2024-04-24T15:59:12.355833+0000","last_active":"2024-04-24T18:25:43.662385+0000","last_peered":"2024-04-24T18:25:43.662385+0000","last_clean":"2024-04-24T18:25:43.662385+0000","last_became_active":"2024-04-24T15:59:12.355529+0000","last_became_peered":"2024-04-24T15:59:12.355529+0000","last_unstale":"2024-04-24T18:25:43.662385+0000","last_undegraded":"2024-04-24T18:25:43.662385+0000","last_fullsized":"2024-04-24T18:25:43.662385+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:45.777687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":2277,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.660936+0000","last_change":"2024-04-24T15:59:09.075498+0000","last_active":"2024-04-24T18:25:43.660936+0000","last_peered":"2024-04-24T18:25:43.660936+0000","last_clean":"2024-04-24T18:25:43.660936+0000","last_became_active":"2024-04-24T15:59:09.074520+0000","last_became_peered":"2024-04-24T15:59:09.074520+0000","last_unstale":"2024-04-24T18:25:43.660936+0000","last_undegraded":"2024-04-24T18:25:43.660936+0000","last_fullsized":"2024-04-24T18:25:43.660936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_clean_scrub_stamp":"2024-04-24T15:59:07.733245+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:20.668162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.506669+0000","last_change":"2024-04-24T15:59:12.358877+0000","last_active":"2024-04-24T18:25:43.506669+0000","last_peered":"2024-04-24T18:25:43.506669+0000","last_clean":"2024-04-24T18:25:43.506669+0000","last_became_active":"2024-04-24T15:59:12.358571+0000","last_became_peered":"2024-04-24T15:59:12.358571+0000","last_unstale":"2024-04-24T18:25:43.506669+0000","last_undegraded":"2024-04-24T18:25:43.506669+0000","last_fullsized":"2024-04-24T18:25:43.506669+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:14:23.463356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.664766+0000","last_change":"2024-04-24T15:59:12.357122+0000","last_active":"2024-04-24T18:25:43.664766+0000","last_peered":"2024-04-24T18:25:43.664766+0000","last_clean":"2024-04-24T18:25:43.664766+0000","last_became_active":"2024-04-24T15:59:12.356793+0000","last_became_peered":"2024-04-24T15:59:12.356793+0000","last_unstale":"2024-04-24T18:25:43.664766+0000","last_undegraded":"2024-04-24T18:25:43.664766+0000","last_fullsized":"2024-04-24T18:25:43.664766+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:35:23.805820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.666429+0000","last_change":"2024-04-24T15:59:12.357605+0000","last_active":"2024-04-24T18:25:43.666429+0000","last_peered":"2024-04-24T18:25:43.666429+0000","last_clean":"2024-04-24T18:25:43.666429+0000","last_became_active":"2024-04-24T15:59:12.357265+0000","last_became_peered":"2024-04-24T15:59:12.357265+0000","last_unstale":"2024-04-24T18:25:43.666429+0000","last_undegraded":"2024-04-24T18:25:43.666429+0000","last_fullsized":"2024-04-24T18:25:43.666429+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:15:21.669877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.667703+0000","last_change":"2024-04-24T15:59:12.357325+0000","last_active":"2024-04-24T18:25:43.667703+0000","last_peered":"2024-04-24T18:25:43.667703+0000","last_clean":"2024-04-24T18:25:43.667703+0000","last_became_active":"2024-04-24T15:59:12.357008+0000","last_became_peered":"2024-04-24T15:59:12.357008+0000","last_unstale":"2024-04-24T18:25:43.667703+0000","last_undegraded":"2024-04-24T18:25:43.667703+0000","last_fullsized":"2024-04-24T18:25:43.667703+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:15:20.941230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949674735,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310460,"kb_used_data":4672,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94061380,"statfs":{"total":96636764160,"available":96318853120,"internally_reserved":0,"allocated":4784128,"data_stored":2570654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.32300000000000001,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":0.48699999999999999,"5min":0.59099999999999997,"15min":2.5350000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.34100000000000003,"15min":0.35799999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":0.42799999999999999,"5min":0.497,"15min":5.782},"last":0.32600000000000001}]},{"osd":1,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47699999999999998,"5min":0.57199999999999995,"15min":6.4219999999999997},"last":0.31},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.28799999999999998,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":0.53200000000000003,"5min":0.65600000000000003,"15min":7.2649999999999997},"last":0.38100000000000001}]}]},{"osd":0,"up_from":9,"seq":38654707441,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306664,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94065176,"statfs":{"total":96636764160,"available":96322740224,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.34499999999999997,"15min":0.34000000000000002},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.46600000000000003,"5min":0.56299999999999994,"15min":1.673},"last":0.432},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.29199999999999998,"5min":0.26300000000000001,"15min":0.23599999999999999},"max":{"1min":0.49299999999999999,"5min":0.54500000000000004,"15min":1.1679999999999999},"last":0.41299999999999998}]},{"osd":2,"last update":"Wed Apr 24 18:25:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34000000000000002,"15min":0.34499999999999997},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.22500000000000001},"max":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.38500000000000001},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.22900000000000001},"max":{"1min":0.50600000000000001,"5min":0.57399999999999995,"15min":0.64200000000000002},"last":0.38}]}]},{"osd":1,"up_from":9,"seq":38654707442,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314612,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94057228,"statfs":{"total":96636764160,"available":96314601472,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.36699999999999999},"min":{"1min":0.249,"5min":0.215,"15min":0.20399999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":8.5389999999999997},"last":0.372},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36499999999999999,"15min":0.442},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.47999999999999998,"5min":0.53900000000000003,"15min":8.5850000000000009},"last":0.34899999999999998}]},{"osd":2,"last update":"Wed Apr 24 18:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39300000000000002,"15min":0.39800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.23300000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":8.7400000000000002},"last":0.318},{"interface":"front","average":{"1min":0.437,"5min":0.40899999999999997,"15min":0.40000000000000002},"min":{"1min":0.33200000000000002,"5min":0.26900000000000002,"15min":0.246},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":6.383},"last":0.39700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:26:36.118 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T18:26:36.466 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T18:26:36.468 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-04-24T18:26:36.481 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":6575,"stamp":"2024-04-24T18:26:35.892119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":931736,"kb_used_data":14128,"kb_used_omap":4,"kb_used_meta":916603,"kb_avail":282183784,"statfs":{"total":289910292480,"available":288956194816,"internally_reserved":0,"allocated":14467072,"data_stored":8892698,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":938601480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001916"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.665727+0000","last_change":"2024-04-24T15:59:12.355878+0000","last_active":"2024-04-24T18:25:43.665727+0000","last_peered":"2024-04-24T18:25:43.665727+0000","last_clean":"2024-04-24T18:25:43.665727+0000","last_became_active":"2024-04-24T15:59:12.355473+0000","last_became_peered":"2024-04-24T15:59:12.355473+0000","last_unstale":"2024-04-24T18:25:43.665727+0000","last_undegraded":"2024-04-24T18:25:43.665727+0000","last_fullsized":"2024-04-24T18:25:43.665727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:37:23.943905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.717028+0000","last_change":"2024-04-24T15:59:12.356273+0000","last_active":"2024-04-24T18:25:43.717028+0000","last_peered":"2024-04-24T18:25:43.717028+0000","last_clean":"2024-04-24T18:25:43.717028+0000","last_became_active":"2024-04-24T15:59:12.355976+0000","last_became_peered":"2024-04-24T15:59:12.355976+0000","last_unstale":"2024-04-24T18:25:43.717028+0000","last_undegraded":"2024-04-24T18:25:43.717028+0000","last_fullsized":"2024-04-24T18:25:43.717028+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:14:20.180867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.509224+0000","last_change":"2024-04-24T15:59:12.357100+0000","last_active":"2024-04-24T18:25:43.509224+0000","last_peered":"2024-04-24T18:25:43.509224+0000","last_clean":"2024-04-24T18:25:43.509224+0000","last_became_active":"2024-04-24T15:59:12.356512+0000","last_became_peered":"2024-04-24T15:59:12.356512+0000","last_unstale":"2024-04-24T18:25:43.509224+0000","last_undegraded":"2024-04-24T18:25:43.509224+0000","last_fullsized":"2024-04-24T18:25:43.509224+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:40.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.662385+0000","last_change":"2024-04-24T15:59:12.355833+0000","last_active":"2024-04-24T18:25:43.662385+0000","last_peered":"2024-04-24T18:25:43.662385+0000","last_clean":"2024-04-24T18:25:43.662385+0000","last_became_active":"2024-04-24T15:59:12.355529+0000","last_became_peered":"2024-04-24T15:59:12.355529+0000","last_unstale":"2024-04-24T18:25:43.662385+0000","last_undegraded":"2024-04-24T18:25:43.662385+0000","last_fullsized":"2024-04-24T18:25:43.662385+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:45.777687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":2277,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.660936+0000","last_change":"2024-04-24T15:59:09.075498+0000","last_active":"2024-04-24T18:25:43.660936+0000","last_peered":"2024-04-24T18:25:43.660936+0000","last_clean":"2024-04-24T18:25:43.660936+0000","last_became_active":"2024-04-24T15:59:09.074520+0000","last_became_peered":"2024-04-24T15:59:09.074520+0000","last_unstale":"2024-04-24T18:25:43.660936+0000","last_undegraded":"2024-04-24T18:25:43.660936+0000","last_fullsized":"2024-04-24T18:25:43.660936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_clean_scrub_stamp":"2024-04-24T15:59:07.733245+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:20.668162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.506669+0000","last_change":"2024-04-24T15:59:12.358877+0000","last_active":"2024-04-24T18:25:43.506669+0000","last_peered":"2024-04-24T18:25:43.506669+0000","last_clean":"2024-04-24T18:25:43.506669+0000","last_became_active":"2024-04-24T15:59:12.358571+0000","last_became_peered":"2024-04-24T15:59:12.358571+0000","last_unstale":"2024-04-24T18:25:43.506669+0000","last_undegraded":"2024-04-24T18:25:43.506669+0000","last_fullsized":"2024-04-24T18:25:43.506669+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:14:23.463356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.664766+0000","last_change":"2024-04-24T15:59:12.357122+0000","last_active":"2024-04-24T18:25:43.664766+0000","last_peered":"2024-04-24T18:25:43.664766+0000","last_clean":"2024-04-24T18:25:43.664766+0000","last_became_active":"2024-04-24T15:59:12.356793+0000","last_became_peered":"2024-04-24T15:59:12.356793+0000","last_unstale":"2024-04-24T18:25:43.664766+0000","last_undegraded":"2024-04-24T18:25:43.664766+0000","last_fullsized":"2024-04-24T18:25:43.664766+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:35:23.805820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.666429+0000","last_change":"2024-04-24T15:59:12.357605+0000","last_active":"2024-04-24T18:25:43.666429+0000","last_peered":"2024-04-24T18:25:43.666429+0000","last_clean":"2024-04-24T18:25:43.666429+0000","last_became_active":"2024-04-24T15:59:12.357265+0000","last_became_peered":"2024-04-24T15:59:12.357265+0000","last_unstale":"2024-04-24T18:25:43.666429+0000","last_undegraded":"2024-04-24T18:25:43.666429+0000","last_fullsized":"2024-04-24T18:25:43.666429+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:15:21.669877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.667703+0000","last_change":"2024-04-24T15:59:12.357325+0000","last_active":"2024-04-24T18:25:43.667703+0000","last_peered":"2024-04-24T18:25:43.667703+0000","last_clean":"2024-04-24T18:25:43.667703+0000","last_became_active":"2024-04-24T15:59:12.357008+0000","last_became_peered":"2024-04-24T15:59:12.357008+0000","last_unstale":"2024-04-24T18:25:43.667703+0000","last_undegraded":"2024-04-24T18:25:43.667703+0000","last_fullsized":"2024-04-24T18:25:43.667703+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:15:20.941230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949674735,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310460,"kb_used_data":4672,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94061380,"statfs":{"total":96636764160,"available":96318853120,"internally_reserved":0,"allocated":4784128,"data_stored":2570654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.32300000000000001,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":0.48699999999999999,"5min":0.59099999999999997,"15min":2.5350000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.34100000000000003,"15min":0.35799999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":0.42799999999999999,"5min":0.497,"15min":5.782},"last":0.32600000000000001}]},{"osd":1,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47699999999999998,"5min":0.57199999999999995,"15min":6.4219999999999997},"last":0.31},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.28799999999999998,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":0.53200000000000003,"5min":0.65600000000000003,"15min":7.2649999999999997},"last":0.38100000000000001}]}]},{"osd":0,"up_from":9,"seq":38654707441,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306664,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94065176,"statfs":{"total":96636764160,"available":96322740224,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.34499999999999997,"15min":0.34000000000000002},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.46600000000000003,"5min":0.56299999999999994,"15min":1.673},"last":0.432},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.29199999999999998,"5min":0.26300000000000001,"15min":0.23599999999999999},"max":{"1min":0.49299999999999999,"5min":0.54500000000000004,"15min":1.1679999999999999},"last":0.41299999999999998}]},{"osd":2,"last update":"Wed Apr 24 18:25:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34000000000000002,"15min":0.34499999999999997},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.22500000000000001},"max":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.38500000000000001},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.22900000000000001},"max":{"1min":0.50600000000000001,"5min":0.57399999999999995,"15min":0.64200000000000002},"last":0.38}]}]},{"osd":1,"up_from":9,"seq":38654707442,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314612,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94057228,"statfs":{"total":96636764160,"available":96314601472,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.36699999999999999},"min":{"1min":0.249,"5min":0.215,"15min":0.20399999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":8.5389999999999997},"last":0.372},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36499999999999999,"15min":0.442},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.47999999999999998,"5min":0.53900000000000003,"15min":8.5850000000000009},"last":0.34899999999999998}]},{"osd":2,"last update":"Wed Apr 24 18:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39300000000000002,"15min":0.39800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.23300000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":8.7400000000000002},"last":0.318},{"interface":"front","average":{"1min":0.437,"5min":0.40899999999999997,"15min":0.40000000000000002},"min":{"1min":0.33200000000000002,"5min":0.26900000000000002,"15min":0.246},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":6.383},"last":0.39700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:26:36.482 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-24T18:26:36.482 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T18:26:36.835 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T18:26:36.838 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-04-24T18:26:36.851 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":6575,"stamp":"2024-04-24T18:26:35.892119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":931736,"kb_used_data":14128,"kb_used_omap":4,"kb_used_meta":916603,"kb_avail":282183784,"statfs":{"total":289910292480,"available":288956194816,"internally_reserved":0,"allocated":14467072,"data_stored":8892698,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":938601480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001916"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.665727+0000","last_change":"2024-04-24T15:59:12.355878+0000","last_active":"2024-04-24T18:25:43.665727+0000","last_peered":"2024-04-24T18:25:43.665727+0000","last_clean":"2024-04-24T18:25:43.665727+0000","last_became_active":"2024-04-24T15:59:12.355473+0000","last_became_peered":"2024-04-24T15:59:12.355473+0000","last_unstale":"2024-04-24T18:25:43.665727+0000","last_undegraded":"2024-04-24T18:25:43.665727+0000","last_fullsized":"2024-04-24T18:25:43.665727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:37:23.943905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.717028+0000","last_change":"2024-04-24T15:59:12.356273+0000","last_active":"2024-04-24T18:25:43.717028+0000","last_peered":"2024-04-24T18:25:43.717028+0000","last_clean":"2024-04-24T18:25:43.717028+0000","last_became_active":"2024-04-24T15:59:12.355976+0000","last_became_peered":"2024-04-24T15:59:12.355976+0000","last_unstale":"2024-04-24T18:25:43.717028+0000","last_undegraded":"2024-04-24T18:25:43.717028+0000","last_fullsized":"2024-04-24T18:25:43.717028+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:14:20.180867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.509224+0000","last_change":"2024-04-24T15:59:12.357100+0000","last_active":"2024-04-24T18:25:43.509224+0000","last_peered":"2024-04-24T18:25:43.509224+0000","last_clean":"2024-04-24T18:25:43.509224+0000","last_became_active":"2024-04-24T15:59:12.356512+0000","last_became_peered":"2024-04-24T15:59:12.356512+0000","last_unstale":"2024-04-24T18:25:43.509224+0000","last_undegraded":"2024-04-24T18:25:43.509224+0000","last_fullsized":"2024-04-24T18:25:43.509224+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:40.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.662385+0000","last_change":"2024-04-24T15:59:12.355833+0000","last_active":"2024-04-24T18:25:43.662385+0000","last_peered":"2024-04-24T18:25:43.662385+0000","last_clean":"2024-04-24T18:25:43.662385+0000","last_became_active":"2024-04-24T15:59:12.355529+0000","last_became_peered":"2024-04-24T15:59:12.355529+0000","last_unstale":"2024-04-24T18:25:43.662385+0000","last_undegraded":"2024-04-24T18:25:43.662385+0000","last_fullsized":"2024-04-24T18:25:43.662385+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:45.777687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":2277,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.660936+0000","last_change":"2024-04-24T15:59:09.075498+0000","last_active":"2024-04-24T18:25:43.660936+0000","last_peered":"2024-04-24T18:25:43.660936+0000","last_clean":"2024-04-24T18:25:43.660936+0000","last_became_active":"2024-04-24T15:59:09.074520+0000","last_became_peered":"2024-04-24T15:59:09.074520+0000","last_unstale":"2024-04-24T18:25:43.660936+0000","last_undegraded":"2024-04-24T18:25:43.660936+0000","last_fullsized":"2024-04-24T18:25:43.660936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_clean_scrub_stamp":"2024-04-24T15:59:07.733245+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:20.668162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.506669+0000","last_change":"2024-04-24T15:59:12.358877+0000","last_active":"2024-04-24T18:25:43.506669+0000","last_peered":"2024-04-24T18:25:43.506669+0000","last_clean":"2024-04-24T18:25:43.506669+0000","last_became_active":"2024-04-24T15:59:12.358571+0000","last_became_peered":"2024-04-24T15:59:12.358571+0000","last_unstale":"2024-04-24T18:25:43.506669+0000","last_undegraded":"2024-04-24T18:25:43.506669+0000","last_fullsized":"2024-04-24T18:25:43.506669+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:14:23.463356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.664766+0000","last_change":"2024-04-24T15:59:12.357122+0000","last_active":"2024-04-24T18:25:43.664766+0000","last_peered":"2024-04-24T18:25:43.664766+0000","last_clean":"2024-04-24T18:25:43.664766+0000","last_became_active":"2024-04-24T15:59:12.356793+0000","last_became_peered":"2024-04-24T15:59:12.356793+0000","last_unstale":"2024-04-24T18:25:43.664766+0000","last_undegraded":"2024-04-24T18:25:43.664766+0000","last_fullsized":"2024-04-24T18:25:43.664766+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:35:23.805820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.666429+0000","last_change":"2024-04-24T15:59:12.357605+0000","last_active":"2024-04-24T18:25:43.666429+0000","last_peered":"2024-04-24T18:25:43.666429+0000","last_clean":"2024-04-24T18:25:43.666429+0000","last_became_active":"2024-04-24T15:59:12.357265+0000","last_became_peered":"2024-04-24T15:59:12.357265+0000","last_unstale":"2024-04-24T18:25:43.666429+0000","last_undegraded":"2024-04-24T18:25:43.666429+0000","last_fullsized":"2024-04-24T18:25:43.666429+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:15:21.669877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.667703+0000","last_change":"2024-04-24T15:59:12.357325+0000","last_active":"2024-04-24T18:25:43.667703+0000","last_peered":"2024-04-24T18:25:43.667703+0000","last_clean":"2024-04-24T18:25:43.667703+0000","last_became_active":"2024-04-24T15:59:12.357008+0000","last_became_peered":"2024-04-24T15:59:12.357008+0000","last_unstale":"2024-04-24T18:25:43.667703+0000","last_undegraded":"2024-04-24T18:25:43.667703+0000","last_fullsized":"2024-04-24T18:25:43.667703+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:15:20.941230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949674735,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310460,"kb_used_data":4672,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94061380,"statfs":{"total":96636764160,"available":96318853120,"internally_reserved":0,"allocated":4784128,"data_stored":2570654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.32300000000000001,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":0.48699999999999999,"5min":0.59099999999999997,"15min":2.5350000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.34100000000000003,"15min":0.35799999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":0.42799999999999999,"5min":0.497,"15min":5.782},"last":0.32600000000000001}]},{"osd":1,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47699999999999998,"5min":0.57199999999999995,"15min":6.4219999999999997},"last":0.31},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.28799999999999998,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":0.53200000000000003,"5min":0.65600000000000003,"15min":7.2649999999999997},"last":0.38100000000000001}]}]},{"osd":0,"up_from":9,"seq":38654707441,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306664,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94065176,"statfs":{"total":96636764160,"available":96322740224,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.34499999999999997,"15min":0.34000000000000002},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.46600000000000003,"5min":0.56299999999999994,"15min":1.673},"last":0.432},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.29199999999999998,"5min":0.26300000000000001,"15min":0.23599999999999999},"max":{"1min":0.49299999999999999,"5min":0.54500000000000004,"15min":1.1679999999999999},"last":0.41299999999999998}]},{"osd":2,"last update":"Wed Apr 24 18:25:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34000000000000002,"15min":0.34499999999999997},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.22500000000000001},"max":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.38500000000000001},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.22900000000000001},"max":{"1min":0.50600000000000001,"5min":0.57399999999999995,"15min":0.64200000000000002},"last":0.38}]}]},{"osd":1,"up_from":9,"seq":38654707442,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314612,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94057228,"statfs":{"total":96636764160,"available":96314601472,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.36699999999999999},"min":{"1min":0.249,"5min":0.215,"15min":0.20399999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":8.5389999999999997},"last":0.372},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36499999999999999,"15min":0.442},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.47999999999999998,"5min":0.53900000000000003,"15min":8.5850000000000009},"last":0.34899999999999998}]},{"osd":2,"last update":"Wed Apr 24 18:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39300000000000002,"15min":0.39800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.23300000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":8.7400000000000002},"last":0.318},{"interface":"front","average":{"1min":0.437,"5min":0.40899999999999997,"15min":0.40000000000000002},"min":{"1min":0.33200000000000002,"5min":0.26900000000000002,"15min":0.246},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":6.383},"last":0.39700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:26:36.852 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-24T18:26:37.201 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T18:26:37.201 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":2155,"fsid":"b9870ca7-a3e3-48c6-875b-287e67643414","created":"2024-04-24T15:58:55.135792+0000","modified":"2024-04-24T18:25:45.507181+0000","last_up_change":"2024-04-24T15:59:06.732126+0000","last_in_change":"2024-04-24T15:58:56.181720+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T15:59:07.567611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-24T15:59:10.739509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"6fc36614-9a61-4a2a-889a-bf72224374da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":40015},{"type":"v1","addr":"172.21.15.42:6809","nonce":40015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":40015},{"type":"v1","addr":"172.21.15.42:6811","nonce":40015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6814","nonce":40015},{"type":"v1","addr":"172.21.15.42:6815","nonce":40015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":40015},{"type":"v1","addr":"172.21.15.42:6813","nonce":40015}]},"public_addr":"172.21.15.42:6809/40015","cluster_addr":"172.21.15.42:6811/40015","heartbeat_back_addr":"172.21.15.42:6815/40015","heartbeat_front_addr":"172.21.15.42:6813/40015","state":["exists","up"]},{"osd":1,"uuid":"bca44732-0beb-4819-9928-ca629336d3d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6816","nonce":40016},{"type":"v1","addr":"172.21.15.42:6817","nonce":40016}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6818","nonce":40016},{"type":"v1","addr":"172.21.15.42:6819","nonce":40016}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6822","nonce":40016},{"type":"v1","addr":"172.21.15.42:6823","nonce":40016}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6820","nonce":40016},{"type":"v1","addr":"172.21.15.42:6821","nonce":40016}]},"public_addr":"172.21.15.42:6817/40016","cluster_addr":"172.21.15.42:6819/40016","heartbeat_back_addr":"172.21.15.42:6823/40016","heartbeat_front_addr":"172.21.15.42:6821/40016","state":["exists","up"]},{"osd":2,"uuid":"56f0f362-3f29-4db1-8086-a02243c2e017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":40000},{"type":"v1","addr":"172.21.15.42:6801","nonce":40000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":40000},{"type":"v1","addr":"172.21.15.42:6803","nonce":40000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":40000},{"type":"v1","addr":"172.21.15.42:6807","nonce":40000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":40000},{"type":"v1","addr":"172.21.15.42:6805","nonce":40000}]},"public_addr":"172.21.15.42:6801/40000","cluster_addr":"172.21.15.42:6803/40000","heartbeat_back_addr":"172.21.15.42:6807/40000","heartbeat_front_addr":"172.21.15.42:6805/40000","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.362853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.366407+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T15:59:01.313848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T18:26:38.218 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-04-24T18:26:38.218 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T18:26:38.386 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-04-24T18:26:38.387 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T18:26:38.387 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-04-24T18:26:38.400 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-04-24T18:26:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T18:26:38.769 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-04-24T18:26:38.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T18:26:38.927 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-04-24T18:26:38.927 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T18:26:38.928 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-04-24T18:26:38.941 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-04-24T18:26:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T18:26:39.313 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-04-24T18:26:39.313 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T18:26:39.472 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-04-24T18:26:39.472 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T18:26:39.472 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-04-24T18:26:39.486 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-04-24T18:26:39.843 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T18:26:39.856 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T18:26:40.209 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T18:26:40.211 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-04-24T18:26:40.224 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":6577,"stamp":"2024-04-24T18:26:39.892834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":931736,"kb_used_data":14128,"kb_used_omap":4,"kb_used_meta":916603,"kb_avail":282183784,"statfs":{"total":289910292480,"available":288956194816,"internally_reserved":0,"allocated":14467072,"data_stored":8892698,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":938601480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001837"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.665727+0000","last_change":"2024-04-24T15:59:12.355878+0000","last_active":"2024-04-24T18:25:43.665727+0000","last_peered":"2024-04-24T18:25:43.665727+0000","last_clean":"2024-04-24T18:25:43.665727+0000","last_became_active":"2024-04-24T15:59:12.355473+0000","last_became_peered":"2024-04-24T15:59:12.355473+0000","last_unstale":"2024-04-24T18:25:43.665727+0000","last_undegraded":"2024-04-24T18:25:43.665727+0000","last_fullsized":"2024-04-24T18:25:43.665727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:37:23.943905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.717028+0000","last_change":"2024-04-24T15:59:12.356273+0000","last_active":"2024-04-24T18:25:43.717028+0000","last_peered":"2024-04-24T18:25:43.717028+0000","last_clean":"2024-04-24T18:25:43.717028+0000","last_became_active":"2024-04-24T15:59:12.355976+0000","last_became_peered":"2024-04-24T15:59:12.355976+0000","last_unstale":"2024-04-24T18:25:43.717028+0000","last_undegraded":"2024-04-24T18:25:43.717028+0000","last_fullsized":"2024-04-24T18:25:43.717028+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:14:20.180867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.509224+0000","last_change":"2024-04-24T15:59:12.357100+0000","last_active":"2024-04-24T18:25:43.509224+0000","last_peered":"2024-04-24T18:25:43.509224+0000","last_clean":"2024-04-24T18:25:43.509224+0000","last_became_active":"2024-04-24T15:59:12.356512+0000","last_became_peered":"2024-04-24T15:59:12.356512+0000","last_unstale":"2024-04-24T18:25:43.509224+0000","last_undegraded":"2024-04-24T18:25:43.509224+0000","last_fullsized":"2024-04-24T18:25:43.509224+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:49:40.681750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.662385+0000","last_change":"2024-04-24T15:59:12.355833+0000","last_active":"2024-04-24T18:25:43.662385+0000","last_peered":"2024-04-24T18:25:43.662385+0000","last_clean":"2024-04-24T18:25:43.662385+0000","last_became_active":"2024-04-24T15:59:12.355529+0000","last_became_peered":"2024-04-24T15:59:12.355529+0000","last_unstale":"2024-04-24T18:25:43.662385+0000","last_undegraded":"2024-04-24T18:25:43.662385+0000","last_fullsized":"2024-04-24T18:25:43.662385+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:45.777687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":2277,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.660936+0000","last_change":"2024-04-24T15:59:09.075498+0000","last_active":"2024-04-24T18:25:43.660936+0000","last_peered":"2024-04-24T18:25:43.660936+0000","last_clean":"2024-04-24T18:25:43.660936+0000","last_became_active":"2024-04-24T15:59:09.074520+0000","last_became_peered":"2024-04-24T15:59:09.074520+0000","last_unstale":"2024-04-24T18:25:43.660936+0000","last_undegraded":"2024-04-24T18:25:43.660936+0000","last_fullsized":"2024-04-24T18:25:43.660936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:07.733245+0000","last_clean_scrub_stamp":"2024-04-24T15:59:07.733245+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:20.668162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2148,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.506669+0000","last_change":"2024-04-24T15:59:12.358877+0000","last_active":"2024-04-24T18:25:43.506669+0000","last_peered":"2024-04-24T18:25:43.506669+0000","last_clean":"2024-04-24T18:25:43.506669+0000","last_became_active":"2024-04-24T15:59:12.358571+0000","last_became_peered":"2024-04-24T15:59:12.358571+0000","last_unstale":"2024-04-24T18:25:43.506669+0000","last_undegraded":"2024-04-24T18:25:43.506669+0000","last_fullsized":"2024-04-24T18:25:43.506669+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:14:23.463356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.664766+0000","last_change":"2024-04-24T15:59:12.357122+0000","last_active":"2024-04-24T18:25:43.664766+0000","last_peered":"2024-04-24T18:25:43.664766+0000","last_clean":"2024-04-24T18:25:43.664766+0000","last_became_active":"2024-04-24T15:59:12.356793+0000","last_became_peered":"2024-04-24T15:59:12.356793+0000","last_unstale":"2024-04-24T18:25:43.664766+0000","last_undegraded":"2024-04-24T18:25:43.664766+0000","last_fullsized":"2024-04-24T18:25:43.664766+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:35:23.805820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.666429+0000","last_change":"2024-04-24T15:59:12.357605+0000","last_active":"2024-04-24T18:25:43.666429+0000","last_peered":"2024-04-24T18:25:43.666429+0000","last_clean":"2024-04-24T18:25:43.666429+0000","last_became_active":"2024-04-24T15:59:12.357265+0000","last_became_peered":"2024-04-24T15:59:12.357265+0000","last_unstale":"2024-04-24T18:25:43.666429+0000","last_undegraded":"2024-04-24T18:25:43.666429+0000","last_fullsized":"2024-04-24T18:25:43.666429+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T22:15:21.669877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2147,"reported_epoch":2154,"state":"active+clean","last_fresh":"2024-04-24T18:25:43.667703+0000","last_change":"2024-04-24T15:59:12.357325+0000","last_active":"2024-04-24T18:25:43.667703+0000","last_peered":"2024-04-24T18:25:43.667703+0000","last_clean":"2024-04-24T18:25:43.667703+0000","last_became_active":"2024-04-24T15:59:12.357008+0000","last_became_peered":"2024-04-24T15:59:12.357008+0000","last_unstale":"2024-04-24T18:25:43.667703+0000","last_undegraded":"2024-04-24T18:25:43.667703+0000","last_fullsized":"2024-04-24T18:25:43.667703+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:15:20.941230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949674736,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310460,"kb_used_data":4672,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94061380,"statfs":{"total":96636764160,"available":96318853120,"internally_reserved":0,"allocated":4784128,"data_stored":2570654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.32300000000000001,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":0.48699999999999999,"5min":0.59099999999999997,"15min":2.5350000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.34100000000000003,"15min":0.35799999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":0.42799999999999999,"5min":0.497,"15min":5.782},"last":0.30499999999999999}]},{"osd":1,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47699999999999998,"5min":0.57199999999999995,"15min":6.4219999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.28799999999999998,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":0.53200000000000003,"5min":0.65600000000000003,"15min":7.2649999999999997},"last":0.32900000000000001}]}]},{"osd":0,"up_from":9,"seq":38654707442,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306664,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94065176,"statfs":{"total":96636764160,"available":96322740224,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.34499999999999997,"15min":0.34000000000000002},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.46600000000000003,"5min":0.56299999999999994,"15min":1.673},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.29199999999999998,"5min":0.26300000000000001,"15min":0.23599999999999999},"max":{"1min":0.49299999999999999,"5min":0.54500000000000004,"15min":1.1679999999999999},"last":0.39300000000000002}]},{"osd":2,"last update":"Wed Apr 24 18:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.22500000000000001},"max":{"1min":0.50800000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.314,"5min":0.29499999999999998,"15min":0.22900000000000001},"max":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.64200000000000002},"last":0.376}]}]},{"osd":1,"up_from":9,"seq":38654707443,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314612,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":309630,"kb_avail":94057228,"statfs":{"total":96636764160,"available":96314601472,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":317061464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.34699999999999998,"15min":0.36699999999999999},"min":{"1min":0.249,"5min":0.215,"15min":0.20399999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":8.5389999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36499999999999999,"15min":0.442},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.47999999999999998,"5min":0.53900000000000003,"15min":8.5850000000000009},"last":0.37}]},{"osd":2,"last update":"Wed Apr 24 18:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39300000000000002,"15min":0.39800000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.23300000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":8.7400000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.40899999999999997,"15min":0.40000000000000002},"min":{"1min":0.33200000000000002,"5min":0.26900000000000002,"15min":0.246},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":6.383},"last":0.43099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:26:40.225 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-24T15:59:11.342400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=11, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=26, tm_sec=37, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:26:40.226 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-24T15:59:11.342400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=11, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=26, tm_sec=37, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:26:40.226 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-24T15:59:11.342400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=11, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=26, tm_sec=37, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:26:40.226 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-24T15:59:11.342400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=11, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=26, tm_sec=37, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:26:40.226 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T15:59:07.733245+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=7, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=26, tm_sec=37, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:26:40.226 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-24T15:59:11.342400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=11, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=26, tm_sec=37, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:26:40.226 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-24T15:59:11.342400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=11, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=26, tm_sec=37, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:26:40.226 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T15:59:11.342400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=11, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=26, tm_sec=37, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:26:40.226 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-24T15:59:11.342400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=11, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=26, tm_sec=37, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:26:40.226 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T18:27:00.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T18:27:00.584 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T18:27:00.587 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-04-24T18:27:00.600 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":6587,"stamp":"2024-04-24T18:26:59.896735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":934856,"kb_used_data":14128,"kb_used_omap":4,"kb_used_meta":920699,"kb_avail":282180664,"statfs":{"total":289910292480,"available":288952999936,"internally_reserved":0,"allocated":14467072,"data_stored":8892698,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":942795784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002434"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2158,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:49.089321+0000","last_change":"2024-04-24T18:26:49.089288+0000","last_active":"2024-04-24T18:26:49.089321+0000","last_peered":"2024-04-24T18:26:49.089321+0000","last_clean":"2024-04-24T18:26:49.089321+0000","last_became_active":"2024-04-24T15:59:12.355473+0000","last_became_peered":"2024-04-24T15:59:12.355473+0000","last_unstale":"2024-04-24T18:26:49.089321+0000","last_undegraded":"2024-04-24T18:26:49.089321+0000","last_fullsized":"2024-04-24T18:26:49.089321+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:49.089179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:49.089179+0000","last_clean_scrub_stamp":"2024-04-24T18:26:49.089179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:38:16.441719+0000","scrub_duration":0.00153305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2159,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:39.273707+0000","last_change":"2024-04-24T18:26:39.273665+0000","last_active":"2024-04-24T18:26:39.273707+0000","last_peered":"2024-04-24T18:26:39.273707+0000","last_clean":"2024-04-24T18:26:39.273707+0000","last_became_active":"2024-04-24T15:59:12.355976+0000","last_became_peered":"2024-04-24T15:59:12.355976+0000","last_unstale":"2024-04-24T18:26:39.273707+0000","last_undegraded":"2024-04-24T18:26:39.273707+0000","last_fullsized":"2024-04-24T18:26:39.273707+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:39.273556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:39.273556+0000","last_clean_scrub_stamp":"2024-04-24T18:26:39.273556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:16.309152+0000","scrub_duration":0.0094647269999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2159,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:41.495062+0000","last_change":"2024-04-24T18:26:41.495022+0000","last_active":"2024-04-24T18:26:41.495062+0000","last_peered":"2024-04-24T18:26:41.495062+0000","last_clean":"2024-04-24T18:26:41.495062+0000","last_became_active":"2024-04-24T15:59:12.356512+0000","last_became_peered":"2024-04-24T15:59:12.356512+0000","last_unstale":"2024-04-24T18:26:41.495062+0000","last_undegraded":"2024-04-24T18:26:41.495062+0000","last_fullsized":"2024-04-24T18:26:41.495062+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:41.494890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:41.494890+0000","last_clean_scrub_stamp":"2024-04-24T18:26:41.494890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:07.732199+0000","scrub_duration":0.020764219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2158,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:55.979445+0000","last_change":"2024-04-24T18:26:55.979416+0000","last_active":"2024-04-24T18:26:55.979445+0000","last_peered":"2024-04-24T18:26:55.979445+0000","last_clean":"2024-04-24T18:26:55.979445+0000","last_became_active":"2024-04-24T15:59:12.355529+0000","last_became_peered":"2024-04-24T15:59:12.355529+0000","last_unstale":"2024-04-24T18:26:55.979445+0000","last_undegraded":"2024-04-24T18:26:55.979445+0000","last_fullsized":"2024-04-24T18:26:55.979445+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:55.979292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:55.979292+0000","last_clean_scrub_stamp":"2024-04-24T18:26:55.979292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:03:33.791086+0000","scrub_duration":0.026626185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"2155'99","reported_seq":2290,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:44.036370+0000","last_change":"2024-04-24T18:26:44.036323+0000","last_active":"2024-04-24T18:26:44.036370+0000","last_peered":"2024-04-24T18:26:44.036370+0000","last_clean":"2024-04-24T18:26:44.036370+0000","last_became_active":"2024-04-24T15:59:09.074520+0000","last_became_peered":"2024-04-24T15:59:09.074520+0000","last_unstale":"2024-04-24T18:26:44.036370+0000","last_undegraded":"2024-04-24T18:26:44.036370+0000","last_fullsized":"2024-04-24T18:26:44.036370+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"2155'99","last_scrub_stamp":"2024-04-24T18:26:44.036204+0000","last_deep_scrub":"2155'99","last_deep_scrub_stamp":"2024-04-24T18:26:44.036204+0000","last_clean_scrub_stamp":"2024-04-24T18:26:44.036204+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:16:05.955468+0000","scrub_duration":0.0057955189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2159,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:40.526797+0000","last_change":"2024-04-24T18:26:40.526763+0000","last_active":"2024-04-24T18:26:40.526797+0000","last_peered":"2024-04-24T18:26:40.526797+0000","last_clean":"2024-04-24T18:26:40.526797+0000","last_became_active":"2024-04-24T15:59:12.358571+0000","last_became_peered":"2024-04-24T15:59:12.358571+0000","last_unstale":"2024-04-24T18:26:40.526797+0000","last_undegraded":"2024-04-24T18:26:40.526797+0000","last_fullsized":"2024-04-24T18:26:40.526797+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:40.526631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:40.526631+0000","last_clean_scrub_stamp":"2024-04-24T18:26:40.526631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:39:59.193013+0000","scrub_duration":0.012335106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2158,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:56.964479+0000","last_change":"2024-04-24T18:26:56.964443+0000","last_active":"2024-04-24T18:26:56.964479+0000","last_peered":"2024-04-24T18:26:56.964479+0000","last_clean":"2024-04-24T18:26:56.964479+0000","last_became_active":"2024-04-24T15:59:12.356793+0000","last_became_peered":"2024-04-24T15:59:12.356793+0000","last_unstale":"2024-04-24T18:26:56.964479+0000","last_undegraded":"2024-04-24T18:26:56.964479+0000","last_fullsized":"2024-04-24T18:26:56.964479+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:56.964320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:56.964320+0000","last_clean_scrub_stamp":"2024-04-24T18:26:56.964320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:21:13.646786+0000","scrub_duration":0.023367056000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2148,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:25:46.614784+0000","last_change":"2024-04-24T15:59:12.357605+0000","last_active":"2024-04-24T18:25:46.614784+0000","last_peered":"2024-04-24T18:25:46.614784+0000","last_clean":"2024-04-24T18:25:46.614784+0000","last_became_active":"2024-04-24T15:59:12.357265+0000","last_became_peered":"2024-04-24T15:59:12.357265+0000","last_unstale":"2024-04-24T18:25:46.614784+0000","last_undegraded":"2024-04-24T18:25:46.614784+0000","last_fullsized":"2024-04-24T18:25:46.614784+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T15:59:11.342400+0000","last_clean_scrub_stamp":"2024-04-24T15:59:11.342400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2158,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:50.094102+0000","last_change":"2024-04-24T18:26:50.094067+0000","last_active":"2024-04-24T18:26:50.094102+0000","last_peered":"2024-04-24T18:26:50.094102+0000","last_clean":"2024-04-24T18:26:50.094102+0000","last_became_active":"2024-04-24T15:59:12.357008+0000","last_became_peered":"2024-04-24T15:59:12.357008+0000","last_unstale":"2024-04-24T18:26:50.094102+0000","last_undegraded":"2024-04-24T18:26:50.094102+0000","last_fullsized":"2024-04-24T18:26:50.094102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:50.093938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:50.093938+0000","last_clean_scrub_stamp":"2024-04-24T18:26:50.093938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:53.641182+0000","scrub_duration":0.031997200000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949674740,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":4672,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":4784128,"data_stored":2570654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.32300000000000001,"15min":0.33300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":0.48699999999999999,"5min":0.59099999999999997,"15min":2.5350000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.34100000000000003,"15min":0.35799999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":0.42799999999999999,"5min":0.497,"15min":5.782},"last":0.34399999999999997}]},{"osd":1,"last update":"Wed Apr 24 18:26:09 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.373,"15min":0.39000000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.47699999999999998,"5min":0.57199999999999995,"15min":6.4219999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.40400000000000003,"15min":0.42099999999999999},"min":{"1min":0.28799999999999998,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":0.53200000000000003,"5min":0.65600000000000003,"15min":7.2649999999999997},"last":0.32600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654707446,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306176,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94065664,"statfs":{"total":96636764160,"available":96323239936,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:25:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.34499999999999997,"15min":0.34000000000000002},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.46600000000000003,"5min":0.56299999999999994,"15min":1.673},"last":0.501},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39900000000000002,"15min":0.38800000000000001},"min":{"1min":0.29199999999999998,"5min":0.26300000000000001,"15min":0.23599999999999999},"max":{"1min":0.49299999999999999,"5min":0.54500000000000004,"15min":1.1679999999999999},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Apr 24 18:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.22500000000000001},"max":{"1min":0.50800000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.314,"5min":0.29499999999999998,"15min":0.22900000000000001},"max":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.64200000000000002},"last":0.55000000000000004}]}]},{"osd":1,"up_from":9,"seq":38654707447,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318464,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":313726,"kb_avail":94053376,"statfs":{"total":96636764160,"available":96310657024,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":321255768},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.34599999999999997,"15min":0.36899999999999999},"min":{"1min":0.246,"5min":0.22700000000000001,"15min":0.21099999999999999},"max":{"1min":0.497,"5min":0.69499999999999995,"15min":8.5389999999999997},"last":0.308},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36399999999999999,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.47099999999999997,"5min":0.53900000000000003,"15min":8.5850000000000009},"last":0.28199999999999997}]},{"osd":2,"last update":"Wed Apr 24 18:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39300000000000002,"15min":0.36899999999999999},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.23300000000000001},"max":{"1min":0.54100000000000004,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.28399999999999997,"5min":0.28000000000000003,"15min":0.25800000000000001},"max":{"1min":0.58799999999999997,"5min":0.74199999999999999,"15min":6.383},"last":0.35599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:27:00.601 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-24T15:59:11.342400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=15, tm_min=59, tm_sec=11, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=18, tm_min=26, tm_sec=37, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T18:27:00.601 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T18:27:20.602 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-24T18:27:20.963 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-04-24T18:27:20.966 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-04-24T18:27:20.979 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":6597,"stamp":"2024-04-24T18:27:19.901496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":934856,"kb_used_data":14128,"kb_used_omap":4,"kb_used_meta":920699,"kb_avail":282180664,"statfs":{"total":289910292480,"available":288952999936,"internally_reserved":0,"allocated":14467072,"data_stored":8892698,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":5112,"internal_metadata":942795784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002897"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":2158,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:49.089321+0000","last_change":"2024-04-24T18:26:49.089288+0000","last_active":"2024-04-24T18:26:49.089321+0000","last_peered":"2024-04-24T18:26:49.089321+0000","last_clean":"2024-04-24T18:26:49.089321+0000","last_became_active":"2024-04-24T15:59:12.355473+0000","last_became_peered":"2024-04-24T15:59:12.355473+0000","last_unstale":"2024-04-24T18:26:49.089321+0000","last_undegraded":"2024-04-24T18:26:49.089321+0000","last_fullsized":"2024-04-24T18:26:49.089321+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:49.089179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:49.089179+0000","last_clean_scrub_stamp":"2024-04-24T18:26:49.089179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:38:16.441719+0000","scrub_duration":0.00153305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2159,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:39.273707+0000","last_change":"2024-04-24T18:26:39.273665+0000","last_active":"2024-04-24T18:26:39.273707+0000","last_peered":"2024-04-24T18:26:39.273707+0000","last_clean":"2024-04-24T18:26:39.273707+0000","last_became_active":"2024-04-24T15:59:12.355976+0000","last_became_peered":"2024-04-24T15:59:12.355976+0000","last_unstale":"2024-04-24T18:26:39.273707+0000","last_undegraded":"2024-04-24T18:26:39.273707+0000","last_fullsized":"2024-04-24T18:26:39.273707+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:39.273556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:39.273556+0000","last_clean_scrub_stamp":"2024-04-24T18:26:39.273556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:31:16.309152+0000","scrub_duration":0.0094647269999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2159,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:41.495062+0000","last_change":"2024-04-24T18:26:41.495022+0000","last_active":"2024-04-24T18:26:41.495062+0000","last_peered":"2024-04-24T18:26:41.495062+0000","last_clean":"2024-04-24T18:26:41.495062+0000","last_became_active":"2024-04-24T15:59:12.356512+0000","last_became_peered":"2024-04-24T15:59:12.356512+0000","last_unstale":"2024-04-24T18:26:41.495062+0000","last_undegraded":"2024-04-24T18:26:41.495062+0000","last_fullsized":"2024-04-24T18:26:41.495062+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:41.494890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:41.494890+0000","last_clean_scrub_stamp":"2024-04-24T18:26:41.494890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:58:07.732199+0000","scrub_duration":0.020764219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2158,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:55.979445+0000","last_change":"2024-04-24T18:26:55.979416+0000","last_active":"2024-04-24T18:26:55.979445+0000","last_peered":"2024-04-24T18:26:55.979445+0000","last_clean":"2024-04-24T18:26:55.979445+0000","last_became_active":"2024-04-24T15:59:12.355529+0000","last_became_peered":"2024-04-24T15:59:12.355529+0000","last_unstale":"2024-04-24T18:26:55.979445+0000","last_undegraded":"2024-04-24T18:26:55.979445+0000","last_fullsized":"2024-04-24T18:26:55.979445+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:55.979292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:55.979292+0000","last_clean_scrub_stamp":"2024-04-24T18:26:55.979292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T21:03:33.791086+0000","scrub_duration":0.026626185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"2155'99","reported_seq":2290,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:44.036370+0000","last_change":"2024-04-24T18:26:44.036323+0000","last_active":"2024-04-24T18:26:44.036370+0000","last_peered":"2024-04-24T18:26:44.036370+0000","last_clean":"2024-04-24T18:26:44.036370+0000","last_became_active":"2024-04-24T15:59:09.074520+0000","last_became_peered":"2024-04-24T15:59:09.074520+0000","last_unstale":"2024-04-24T18:26:44.036370+0000","last_undegraded":"2024-04-24T18:26:44.036370+0000","last_fullsized":"2024-04-24T18:26:44.036370+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"2155'99","last_scrub_stamp":"2024-04-24T18:26:44.036204+0000","last_deep_scrub":"2155'99","last_deep_scrub_stamp":"2024-04-24T18:26:44.036204+0000","last_clean_scrub_stamp":"2024-04-24T18:26:44.036204+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:16:05.955468+0000","scrub_duration":0.0057955189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2159,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:40.526797+0000","last_change":"2024-04-24T18:26:40.526763+0000","last_active":"2024-04-24T18:26:40.526797+0000","last_peered":"2024-04-24T18:26:40.526797+0000","last_clean":"2024-04-24T18:26:40.526797+0000","last_became_active":"2024-04-24T15:59:12.358571+0000","last_became_peered":"2024-04-24T15:59:12.358571+0000","last_unstale":"2024-04-24T18:26:40.526797+0000","last_undegraded":"2024-04-24T18:26:40.526797+0000","last_fullsized":"2024-04-24T18:26:40.526797+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:40.526631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:40.526631+0000","last_clean_scrub_stamp":"2024-04-24T18:26:40.526631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:39:59.193013+0000","scrub_duration":0.012335106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2158,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:56.964479+0000","last_change":"2024-04-24T18:26:56.964443+0000","last_active":"2024-04-24T18:26:56.964479+0000","last_peered":"2024-04-24T18:26:56.964479+0000","last_clean":"2024-04-24T18:26:56.964479+0000","last_became_active":"2024-04-24T15:59:12.356793+0000","last_became_peered":"2024-04-24T15:59:12.356793+0000","last_unstale":"2024-04-24T18:26:56.964479+0000","last_undegraded":"2024-04-24T18:26:56.964479+0000","last_fullsized":"2024-04-24T18:26:56.964479+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:56.964320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:56.964320+0000","last_clean_scrub_stamp":"2024-04-24T18:26:56.964320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:21:13.646786+0000","scrub_duration":0.023367056000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2158,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:27:05.840546+0000","last_change":"2024-04-24T18:27:05.840511+0000","last_active":"2024-04-24T18:27:05.840546+0000","last_peered":"2024-04-24T18:27:05.840546+0000","last_clean":"2024-04-24T18:27:05.840546+0000","last_became_active":"2024-04-24T15:59:12.357265+0000","last_became_peered":"2024-04-24T15:59:12.357265+0000","last_unstale":"2024-04-24T18:27:05.840546+0000","last_undegraded":"2024-04-24T18:27:05.840546+0000","last_fullsized":"2024-04-24T18:27:05.840546+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:27:05.840361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:27:05.840361+0000","last_clean_scrub_stamp":"2024-04-24T18:27:05.840361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:55:04.041541+0000","scrub_duration":0.010979694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2158,"reported_epoch":2155,"state":"active+clean","last_fresh":"2024-04-24T18:26:50.094102+0000","last_change":"2024-04-24T18:26:50.094067+0000","last_active":"2024-04-24T18:26:50.094102+0000","last_peered":"2024-04-24T18:26:50.094102+0000","last_clean":"2024-04-24T18:26:50.094102+0000","last_became_active":"2024-04-24T15:59:12.357008+0000","last_became_peered":"2024-04-24T15:59:12.357008+0000","last_unstale":"2024-04-24T18:26:50.094102+0000","last_undegraded":"2024-04-24T18:26:50.094102+0000","last_fullsized":"2024-04-24T18:26:50.094102+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T18:26:50.093938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T18:26:50.093938+0000","last_clean_scrub_stamp":"2024-04-24T18:26:50.093938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:57:53.641182+0000","scrub_duration":0.031997200000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6728,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949674744,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":4672,"kb_used_omap":1,"kb_used_meta":305534,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":4784128,"data_stored":2570654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312867160},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.32500000000000001,"15min":0.33500000000000002},"min":{"1min":0.25,"5min":0.22800000000000001,"15min":0.19500000000000001},"max":{"1min":0.47799999999999998,"5min":0.48699999999999999,"15min":2.5350000000000001},"last":0.315},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33900000000000002,"15min":0.35899999999999999},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.22700000000000001},"max":{"1min":0.46500000000000002,"5min":0.497,"15min":5.782},"last":0.36199999999999999}]},{"osd":1,"last update":"Wed Apr 24 18:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36699999999999999,"15min":0.39200000000000002},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.48299999999999998,"5min":0.51900000000000002,"15min":6.4219999999999997},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.40200000000000002,"15min":0.42099999999999999},"min":{"1min":0.308,"5min":0.27200000000000002,"15min":0.26700000000000002},"max":{"1min":0.53700000000000003,"5min":0.57299999999999995,"15min":7.2649999999999997},"last":0.41699999999999998}]}]},{"osd":0,"up_from":9,"seq":38654707450,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306176,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":301438,"kb_avail":94065664,"statfs":{"total":96636764160,"available":96323239936,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":308672856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Apr 24 18:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.35099999999999998,"15min":0.34200000000000003},"min":{"1min":0.26900000000000002,"5min":0.23499999999999999,"15min":0.21299999999999999},"max":{"1min":0.53000000000000003,"5min":0.56299999999999994,"15min":1.673},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39700000000000002,"15min":0.38900000000000001},"min":{"1min":0.29199999999999998,"5min":0.26300000000000001,"15min":0.23599999999999999},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":1.1679999999999999},"last":0.27400000000000002}]},{"osd":2,"last update":"Wed Apr 24 18:26:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.33800000000000002,"15min":0.34499999999999997},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.22500000000000001},"max":{"1min":0.50800000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.314,"5min":0.29499999999999998,"15min":0.22900000000000001},"max":{"1min":0.51300000000000001,"5min":0.57399999999999995,"15min":0.64200000000000002},"last":0.374}]}]},{"osd":1,"up_from":9,"seq":38654707451,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318464,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":313726,"kb_avail":94053376,"statfs":{"total":96636764160,"available":96310657024,"internally_reserved":0,"allocated":4841472,"data_stored":3161022,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1704,"internal_metadata":321255768},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Apr 24 18:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.34599999999999997,"15min":0.36899999999999999},"min":{"1min":0.246,"5min":0.22700000000000001,"15min":0.21099999999999999},"max":{"1min":0.497,"5min":0.69499999999999995,"15min":8.5389999999999997},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36399999999999999,"15min":0.41599999999999998},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.47099999999999997,"5min":0.53900000000000003,"15min":8.5850000000000009},"last":0.30399999999999999}]},{"osd":2,"last update":"Wed Apr 24 18:26:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39300000000000002,"15min":0.36899999999999999},"min":{"1min":0.314,"5min":0.25900000000000001,"15min":0.23300000000000001},"max":{"1min":0.54100000000000004,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.28399999999999997,"5min":0.28000000000000003,"15min":0.25800000000000001},"max":{"1min":0.58799999999999997,"5min":0.74199999999999999,"15min":6.383},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3364,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T18:27:20.981 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-04-24T18:27:21.352 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T18:27:21.353 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T18:27:21.353 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-04-24T18:27:21.353 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T18:27:21.411 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-04-24T18:27:21.412 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-04-24T18:27:21.412 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T18:27:21.464 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-04-24T18:27:21.464 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-04-24T18:27:21.465 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T18:27:21.518 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-04-24T18:27:21.519 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T18:27:21.519 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-04-24T18:27:21.519 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T18:27:21.546 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-04-24T18:27:21.546 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T18:27:21.547 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-04-24T18:27:21.547 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T18:27:21.606 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-04-24T18:27:21.606 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T18:27:21.607 DEBUG:teuthology.orchestra.run.smithi042:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-04-24T18:27:21.682 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi042.front.sepia.ceph.com 2024-04-24T18:27:21.682 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-04-24T18:27:21.902 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi042.front.sepia.ceph.com 2024-04-24T18:27:21.902 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-04-24T18:27:22.014 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi042.front.sepia.ceph.com 2024-04-24T18:27:22.014 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-04-24T18:27:22.111 INFO:tasks.ceph:Archiving mon data... 2024-04-24T18:27:22.112 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592/data/cluster1.mon.a.tgz 2024-04-24T18:27:22.112 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-04-24T18:27:22.133 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.bcgqtvwkfT 2024-04-24T18:27:22.133 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.bcgqtvwkfT 2024-04-24T18:27:22.318 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.bcgqtvwkfT 2024-04-24T18:27:22.409 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.bcgqtvwkfT is 586KB 2024-04-24T18:27:22.508 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.bcgqtvwkfT 2024-04-24T18:27:22.527 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T18:27:22.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T18:27:22.571 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T18:27:22.604 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-24T18:27:22.604 DEBUG:teuthology.orchestra.run.smithi042:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-24T18:27:22.644 DEBUG:teuthology.orchestra.run.smithi136:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-24T18:27:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-24T18:27:22.676 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-24T18:27:22.695 INFO:tasks.ceph:Archiving crash dumps... 2024-04-24T18:27:22.697 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592/remote/smithi042/crash 2024-04-24T18:27:22.698 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T18:27:22.752 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592/remote/smithi136/crash 2024-04-24T18:27:22.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T18:27:22.783 INFO:tasks.ceph:Compressing logs... 2024-04-24T18:27:22.783 DEBUG:teuthology.orchestra.run.smithi042:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T18:27:22.797 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T18:27:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43190.log 2024-04-24T18:27:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43489.log 2024-04-24T18:27:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48561.log 2024-04-24T18:27:22.829 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43190.log: /var/log/ceph/cluster1-client.admin.43489.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43190.log.gz 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43189.log 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43489.log.gz 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43583.log 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48561.log.gz 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43744.log 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43189.log.gz 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43421.log 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43583.log.gz 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48469.log 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43744.log.gz 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48285.log 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43421.log.gz 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-04-24T18:27:22.830 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48469.log.gz 2024-04-24T18:27:22.831 INFO:teuthology.orchestra.run.smithi042.stderr:gzip /var/log/ceph/cluster1-client.admin.48285.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.43652.log 2024-04-24T18:27:22.831 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48285.log.gz 2024-04-24T18:27:22.831 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.34961.log 2024-04-24T18:27:22.831 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43282.log 2024-04-24T18:27:22.831 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43652.log.gz 2024-04-24T18:27:22.831 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph.tmp-client.admin.34961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-04-24T18:27:22.831 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.34961.log.gz 2024-04-24T18:27:22.831 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43282.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43282.log.gz 2024-04-24T18:27:22.831 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43052.log 2024-04-24T18:27:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43328.log 2024-04-24T18:27:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43052.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43052.log.gz 2024-04-24T18:27:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48193.log 2024-04-24T18:27:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43098.log 2024-04-24T18:27:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43328.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43328.log.gz 2024-04-24T18:27:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48331.log 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48193.log.gz 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48377.log 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43098.log.gz 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40079.log 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48331.log.gz 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48377.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48009.log 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48377.log.gz 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.40079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48607.log 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40079.log.gz 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48009.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48009.log.gz 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39997.log 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48607.log.gz 2024-04-24T18:27:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39834.log 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.39997.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39997.log.gz 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.39834.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39834.log.gz 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48147.log 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48101.log 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39788.log 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48147.log.gz 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48101.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48101.log.gz 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.39788.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39788.log.gz 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48515.log 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43560.log 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48651.log 2024-04-24T18:27:22.838 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48515.log.gz 2024-04-24T18:27:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43560.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43560.log.gz 2024-04-24T18:27:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48651.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48651.log.gz 2024-04-24T18:27:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-04-24T18:27:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39616.log 2024-04-24T18:27:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43446.log 2024-04-24T18:27:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.39616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39616.log.gz 2024-04-24T18:27:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43446.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43446.log.gz 2024-04-24T18:27:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42851.log 2024-04-24T18:27:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48239.log 2024-04-24T18:27:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.42851.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42851.log.gz 2024-04-24T18:27:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48239.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48239.log.gz 2024-04-24T18:27:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43327.log 2024-04-24T18:27:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-04-24T18:27:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43327.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43327.log.gz 2024-04-24T18:27:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-osd.2.log: 94.2% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-04-24T18:27:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-04-24T18:27:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48055.log 2024-04-24T18:27:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.48055.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48055.log.gz 2024-04-24T18:27:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42699.log 2024-04-24T18:27:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.42699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42699.log.gz 2024-04-24T18:27:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43006.log 2024-04-24T18:27:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43006.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43006.log.gz 2024-04-24T18:27:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-04-24T18:27:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48423.log 2024-04-24T18:27:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.48423.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48423.log.gz 2024-04-24T18:27:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42914.log 2024-04-24T18:27:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.42914.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42914.log.gz 2024-04-24T18:27:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43188.log 2024-04-24T18:27:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43188.log.gz 2024-04-24T18:27:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39881.log 2024-04-24T18:27:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.39881.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39881.log.gz 2024-04-24T18:27:22.859 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43698.log 2024-04-24T18:27:22.860 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.43698.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43698.log.gz 2024-04-24T18:27:22.860 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-04-24T18:27:22.870 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-mon.a.log: 86.6% -- replaced with /var/log/ceph/cluster1.log.gz 2024-04-24T18:27:22.870 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42960.log 2024-04-24T18:27:22.870 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/cluster1-client.admin.42960.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42960.log.gz 2024-04-24T18:27:22.871 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58064.log 2024-04-24T18:27:22.871 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53933.log 2024-04-24T18:27:22.871 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.58064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44326.log 2024-04-24T18:27:22.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65185.log 2024-04-24T18:27:22.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60396.log 2024-04-24T18:27:22.873 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44326.log.gz 2024-04-24T18:27:22.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.65185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54942.log 2024-04-24T18:27:22.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.60396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71088.log 2024-04-24T18:27:22.873 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.60396.log.gz 2024-04-24T18:27:22.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64755.log 2024-04-24T18:27:22.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71088.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.65185.log.gz 2024-04-24T18:27:22.874 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65424.log 2024-04-24T18:27:22.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65271.log 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71088.log.gz 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52471.log 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.65424.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65424.log.gz 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.65271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44211.log 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64755.log.gz 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52471.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.65271.log.gz 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49689.log 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51191.log 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44211.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44211.log.gz 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63025.log 2024-04-24T18:27:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.51191.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47474.log 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.63025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45985.log 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46967.log 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.45985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68516.log 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63025.log.gz 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53347.log 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.68516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68376.log 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52901.log 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.68376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60053.log 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.68376.log.gz 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45799.log 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60053.log: /var/log/ceph/cluster1-client.admin.50337.log 2024-04-24T18:27:22.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.45799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57676.log 2024-04-24T18:27:22.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.50337.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71193.log 2024-04-24T18:27:22.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69777.log 2024-04-24T18:27:22.883 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70953.log 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52349.log 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71193.log.gz 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52712.log 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69777.log.gz 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.52349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47213.log 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.52349.log.gz 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59772.log 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70953.log.gz 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.47213.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.47213.log.gzgzip 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.56736.log 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51654.log 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55840.log 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59069.log 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55840.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65374.log 2024-04-24T18:27:22.884 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.59069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56016.log 2024-04-24T18:27:22.885 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.65374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59048.log 2024-04-24T18:27:22.885 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58064.log.gz 2024-04-24T18:27:22.885 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.56016.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.65374.log.gz 2024-04-24T18:27:22.885 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50234.log 2024-04-24T18:27:22.885 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.59048.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48937.log 2024-04-24T18:27:22.885 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.50234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53056.log 2024-04-24T18:27:22.885 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45225.log 2024-04-24T18:27:22.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.53056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59276.log 2024-04-24T18:27:22.888 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64671.log 2024-04-24T18:27:22.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.45225.log: 89.8% 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52901.log.gz 2024-04-24T18:27:22.888 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.45225.log.gz 2024-04-24T18:27:22.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.59276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53996.log 2024-04-24T18:27:22.888 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45592.log 2024-04-24T18:27:22.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64671.log: /var/log/ceph/cluster2-client.mirror.53996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68483.log 2024-04-24T18:27:22.888 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.45592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57125.log 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71130.log 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64671.log.gz 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57125.log: /var/log/ceph/cluster1-client.mirror.71130.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68483.log.gz 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53056.log.gz 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71130.log.gz 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57484.log 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62616.log 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.57484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49156.log 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64044.log 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48937.log.gz 2024-04-24T18:27:22.889 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.49156.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48266.log 2024-04-24T18:27:22.895 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45133.log 2024-04-24T18:27:22.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.64044.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62616.log.gz 2024-04-24T18:27:22.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.48266.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61714.log 2024-04-24T18:27:22.895 INFO:teuthology.orchestra.run.smithi136.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59276.log.gz 2024-04-24T18:27:22.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.45133.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64044.log.gz 2024-04-24T18:27:22.895 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.45133.log.gz 2024-04-24T18:27:22.895 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69051.log 2024-04-24T18:27:22.895 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70059.log 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50920.log 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61714.log.gz 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48740.log 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46581.log 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69051.log.gz 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48740.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70059.log.gz 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62549.log 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46707.log 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49001.log 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59048.log.gz 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60859.log 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.49001.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62549.log.gz 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61310.log 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59069.log.gz 2024-04-24T18:27:22.896 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64297.log 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60859.log: /var/log/ceph/cluster1-client.admin.61310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63362.log 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.64297.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59357.log 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63362.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.64297.log.gz 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64487.log 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.59357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61953.log 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64487.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71627.log 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63362.log.gz 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50208.log 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.71627.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64487.log.gz 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster2-client.admin.71627.log.gz 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61953.log.gz 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.50208.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.50208.log.gz 2024-04-24T18:27:22.897 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46539.log 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69030.log 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46757.log 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70236.log 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46757.log: /var/log/ceph/cluster2-client.mirror.70236.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69030.log.gz 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54551.log 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57043.log 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54551.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70236.log.gz 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60640.log 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46318.log 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55193.log 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55280.log 2024-04-24T18:27:22.901 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.55193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51355.log 2024-04-24T18:27:22.902 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.55280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55439.log 2024-04-24T18:27:22.902 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.51355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45833.log 2024-04-24T18:27:22.902 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.51355.log.gz 2024-04-24T18:27:22.902 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66736.log 2024-04-24T18:27:22.902 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.45833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53652.log 2024-04-24T18:27:22.902 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48219.log 2024-04-24T18:27:22.902 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69221.log 2024-04-24T18:27:22.902 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.48219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54509.log 2024-04-24T18:27:22.902 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66736.log.gz 2024-04-24T18:27:22.902 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.48219.log.gz 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63222.log 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54509.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72399.log 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.63222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56672.log 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69221.log.gz 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64621.log 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72399.log.gz 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59969.log 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56736.log.gz 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46497.log 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63222.log.gz 2024-04-24T18:27:22.906 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59969.log: /var/log/ceph/cluster1-client.mirror.46497.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64621.log.gz 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54214.log 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45945.log 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.54214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63218.log 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53933.log.gz 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.45945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58332.log 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.63218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55621.log 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63218.log.gz 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.58332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46728.log 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53996.log.gz 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55621.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68093.log 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46728.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65547.log 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69615.log 2024-04-24T18:27:22.907 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68768.log 2024-04-24T18:27:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69615.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45782.log 2024-04-24T18:27:22.910 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68093.log.gz 2024-04-24T18:27:22.910 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65547.log.gz 2024-04-24T18:27:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57803.log 2024-04-24T18:27:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.45782.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69615.log.gz 2024-04-24T18:27:22.910 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57125.log.gz 2024-04-24T18:27:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68768.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68768.log.gz 2024-04-24T18:27:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50658.log 2024-04-24T18:27:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52683.log 2024-04-24T18:27:22.910 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58963.log 2024-04-24T18:27:22.911 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53778.log 2024-04-24T18:27:22.911 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64372.log 2024-04-24T18:27:22.911 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62253.log 2024-04-24T18:27:22.911 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.64372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47062.log 2024-04-24T18:27:22.911 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64372.log.gz 2024-04-24T18:27:22.911 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60053.log.gz 2024-04-24T18:27:22.911 INFO:teuthology.orchestra.run.smithi136.stderr:gzip /var/log/ceph/cluster2-client.admin.62253.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58985.log 2024-04-24T18:27:22.911 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.62253.log.gz 2024-04-24T18:27:22.911 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.47062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55048.log 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62827.log 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr:gzip /var/log/ceph/cluster1-client.mirror.55048.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56037.log 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62700.log 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.56037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67450.log 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62700.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62827.log.gz 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60775.log 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67450.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62130.log 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62700.log.gz 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53454.log 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.62130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54367.log 2024-04-24T18:27:22.914 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67450.log.gz 2024-04-24T18:27:22.915 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44441.log 2024-04-24T18:27:22.915 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.54367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71244.log 2024-04-24T18:27:22.915 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62130.log.gz 2024-04-24T18:27:22.915 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54109.log 2024-04-24T18:27:22.915 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44441.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44441.log.gz 2024-04-24T18:27:22.915 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40506.log 2024-04-24T18:27:22.915 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.54109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51832.log 2024-04-24T18:27:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.40506.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71244.log.gz 2024-04-24T18:27:22.919 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49773.log 2024-04-24T18:27:22.919 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40506.log.gz 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51684.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46341.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69975.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53190.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52515.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.53190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46468.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53190.log.gz 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52515.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64214.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69975.log.gz 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56354.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.64214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47912.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48719.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57189.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55189.log 2024-04-24T18:27:22.920 INFO:teuthology.orchestra.run.smithi136.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64214.log.gz 2024-04-24T18:27:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55393.log 2024-04-24T18:27:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.55189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47084.log 2024-04-24T18:27:22.921 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55189.log.gz 2024-04-24T18:27:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.55393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58085.log 2024-04-24T18:27:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.47084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44579.log 2024-04-24T18:27:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69301.log 2024-04-24T18:27:22.921 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.44579.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44579.log.gz 2024-04-24T18:27:22.921 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55840.log.gz 2024-04-24T18:27:22.922 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69301.log: /var/log/ceph/cluster2-client.mirror.58085.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.69301.log.gz 2024-04-24T18:27:22.922 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59141.log 2024-04-24T18:27:22.922 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48613.log 2024-04-24T18:27:22.922 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 /var/log/ceph/cluster1-client.mirror.59141.log: --verbose -- /var/log/ceph/cluster1-client.mirror.71215.log 2024-04-24T18:27:22.922 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66714.log 2024-04-24T18:27:22.923 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56016.log.gz 2024-04-24T18:27:22.923 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66188.log 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53484.log 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.66188.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68403.log 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71215.log.gz 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66714.log.gz 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64002.log 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.68403.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66188.log.gz 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56079.log 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.64002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71067.log 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.68403.log.gz 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.56079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44072.log 2024-04-24T18:27:22.927 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64002.log.gz 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49844.log 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49156.log.gz 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68697.log 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44072.log.gz 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49844.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71067.log.gz 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58807.log 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46560.log 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62410.log 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr: 67.2% -- replaced with /var/log/ceph/cluster2-client.admin.45799.log.gz 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:gzip /var/log/ceph/cluster1-client.mirror.46560.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51727.log 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68697.log.gz 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.62410.log: 18.1% -- replaced with /var/log/ceph/cluster1-client.admin.62410.log.gz 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52420.log 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52557.log 2024-04-24T18:27:22.928 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51727.log: /var/log/ceph/cluster1-client.mirror.52420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53014.log 2024-04-24T18:27:22.929 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64952.log 2024-04-24T18:27:22.929 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.53014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62195.log 2024-04-24T18:27:22.933 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64023.log 2024-04-24T18:27:22.933 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64952.log: /var/log/ceph/cluster2-client.mirror.62195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56164.log 2024-04-24T18:27:22.933 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.64023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62941.log 2024-04-24T18:27:22.933 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64952.log.gz 2024-04-24T18:27:22.933 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62195.log.gz 2024-04-24T18:27:22.933 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.56164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56100.log 2024-04-24T18:27:22.933 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64023.log.gz 2024-04-24T18:27:22.933 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49794.log 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.50337.log.gz 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.56100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56387.log 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72169.log 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.56387.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62941.log.gz 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51307.log 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64973.log 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72169.log.gz 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.51307.log: 54.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45592.log.gz 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53715.log 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65400.log 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58156.log 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64267.log 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64973.log.gz 2024-04-24T18:27:22.934 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.58156.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55382.log 2024-04-24T18:27:22.935 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51416.log 2024-04-24T18:27:22.935 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.64267.log: /var/log/ceph/cluster2-client.admin.55382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56610.log 2024-04-24T18:27:22.935 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65400.log.gz 2024-04-24T18:27:22.935 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.55382.log.gz 2024-04-24T18:27:22.935 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.64267.log.gz 2024-04-24T18:27:22.935 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66934.log 2024-04-24T18:27:22.935 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51811.log 2024-04-24T18:27:22.935 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70483.log 2024-04-24T18:27:22.935 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.45974.log 2024-04-24T18:27:22.938 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.70483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58921.log 2024-04-24T18:27:22.938 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50920.log.gz 2024-04-24T18:27:22.938 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66934.log.gz 2024-04-24T18:27:22.938 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.0.45974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40316.log 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.70483.log.gz 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61777.log 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.40316.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40316.log.gz 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64881.log 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54808.log 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64466.log 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54808.log: gzip 89.8%-5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.61777.log.gz-- 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.52030.log 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58611.log 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49689.log.gz 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64881.log.gz 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.52030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69137.log 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64466.log.gz 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43751.log 2024-04-24T18:27:22.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69137.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66248.log 2024-04-24T18:27:22.940 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.43751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67727.log 2024-04-24T18:27:22.940 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43751.log.gz 2024-04-24T18:27:22.940 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.66248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67769.log 2024-04-24T18:27:22.940 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.66248.log.gz 2024-04-24T18:27:22.940 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69137.log.gz 2024-04-24T18:27:22.940 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45528.log 2024-04-24T18:27:22.940 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53347.log.gz 2024-04-24T18:27:22.940 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63118.log 2024-04-24T18:27:22.940 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.45528.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67727.log.gz 2024-04-24T18:27:22.940 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47693.log 2024-04-24T18:27:22.943 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.63118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61036.log 2024-04-24T18:27:22.943 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67769.log.gz 2024-04-24T18:27:22.943 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64931.log 2024-04-24T18:27:22.943 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63118.log.gz 2024-04-24T18:27:22.943 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.61036.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47757.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53652.log.gz 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59462.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55140.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59243.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64931.log.gz 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.55140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63605.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.59243.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52051.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59416.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.52051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65120.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.59416.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67959.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63605.log.gz 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.65120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46162.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55307.log 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46162.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65120.log.gz 2024-04-24T18:27:22.944 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68030.log 2024-04-24T18:27:22.945 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.55307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71328.log 2024-04-24T18:27:22.945 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67959.log.gz 2024-04-24T18:27:22.945 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54787.log 2024-04-24T18:27:22.949 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51025.log 2024-04-24T18:27:22.949 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62037.log 2024-04-24T18:27:22.949 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68030.log.gz 2024-04-24T18:27:22.949 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.51025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66891.log 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57676.log.gz 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.62037.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71328.log.gz 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65435.log 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50555.log 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.65435.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62037.log.gz 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69912.log 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66891.log.gz 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53380.log 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44947.log 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.53380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50616.log 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69912.log.gz 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.44947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47312.log 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44947.log.gz 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52199.log 2024-04-24T18:27:22.950 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68718.log 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.52199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57634.log 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr:gzip /var/log/ceph/cluster1-client.mirror.68718.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56477.log 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58828.log 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68718.log.gz 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56693.log 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50828.log 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61414.log 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61911.log 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50828.log: /var/log/ceph/cluster2-client.admin.61414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61974.log 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr: 18.7% -- replaced with /var/log/ceph/cluster2-client.admin.61414.log.gz 2024-04-24T18:27:22.951 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51226.log 2024-04-24T18:27:22.953 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44717.log 2024-04-24T18:27:22.953 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.51226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62785.log 2024-04-24T18:27:22.953 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61911.log.gz 2024-04-24T18:27:22.953 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61974.log.gz 2024-04-24T18:27:22.953 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44717.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44717.log.gz 2024-04-24T18:27:22.953 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55027.log 2024-04-24T18:27:22.953 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58879.log 2024-04-24T18:27:22.953 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.55027.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68597.log 2024-04-24T18:27:22.953 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58879.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62785.log.gz 2024-04-24T18:27:22.953 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68789.log 2024-04-24T18:27:22.954 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68789.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68597.log.gz 2024-04-24T18:27:22.954 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54808.log.gz 2024-04-24T18:27:22.954 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68789.log.gz 2024-04-24T18:27:22.954 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53196.log 2024-04-24T18:27:22.955 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69480.log 2024-04-24T18:27:22.955 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.53196.log: /var/log/ceph/cluster2-client.mirror.69480.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54942.log.gz 2024-04-24T18:27:22.955 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69480.log.gz 2024-04-24T18:27:22.955 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68177.log 2024-04-24T18:27:22.956 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49584.log 2024-04-24T18:27:22.956 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54900.log 2024-04-24T18:27:22.957 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49584.log: /var/log/ceph/cluster1-client.mirror.54900.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55048.log.gz 2024-04-24T18:27:22.957 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45867.log 2024-04-24T18:27:22.957 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68177.log.gz 2024-04-24T18:27:22.957 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49072.log 2024-04-24T18:27:22.958 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.45867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58262.log 2024-04-24T18:27:22.958 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.49072.log: /var/log/ceph/cluster2-client.mirror.58262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43705.log 2024-04-24T18:27:22.959 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55140.log.gz 2024-04-24T18:27:22.959 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60137.log 2024-04-24T18:27:22.959 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.43705.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43705.log.gz 2024-04-24T18:27:22.959 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54263.log 2024-04-24T18:27:22.959 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55439.log.gz 2024-04-24T18:27:22.961 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.60137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59112.log 2024-04-24T18:27:22.961 INFO:teuthology.orchestra.run.smithi136.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55193.log.gz 2024-04-24T18:27:22.961 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.54263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54660.log 2024-04-24T18:27:22.961 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53849.log 2024-04-24T18:27:22.961 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.59112.log: /var/log/ceph/cluster1-client.mirror.54660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54290.log 2024-04-24T18:27:22.961 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46184.log 2024-04-24T18:27:22.961 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53849.log: /var/log/ceph/cluster1-client.admin.54290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68009.log 2024-04-24T18:27:22.961 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.46184.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55663.log 2024-04-24T18:27:22.962 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46184.log.gz 2024-04-24T18:27:22.962 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.45945.log 2024-04-24T18:27:22.963 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55663.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46883.log 2024-04-24T18:27:22.963 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68009.log.gz 2024-04-24T18:27:22.963 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.0.45945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68114.log 2024-04-24T18:27:22.963 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72261.log 2024-04-24T18:27:22.964 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51591.log 2024-04-24T18:27:22.964 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72261.log: gzip -5 --verbose -- 90.5%/var/log/ceph/cluster2-client.mirror.52008.log 2024-04-24T18:27:22.964 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51191.log.gz 2024-04-24T18:27:22.964 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72261.log.gz 2024-04-24T18:27:22.964 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51591.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68114.log.gz 2024-04-24T18:27:22.964 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.52008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45765.log 2024-04-24T18:27:22.964 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66693.log 2024-04-24T18:27:22.964 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51654.log.gz 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63584.log 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.45765.log: /var/log/ceph/cluster1-client.mirror.66693.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51727.log.gz 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56058.log 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62333.log 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66693.log.gz 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.56058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72023.log 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63584.log.gz 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.62333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66799.log 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60032.log 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.62333.log.gz 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72023.log.gz 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53694.log 2024-04-24T18:27:22.967 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.60032.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48223.log 2024-04-24T18:27:22.970 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53694.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62279.log 2024-04-24T18:27:22.970 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66799.log.gz 2024-04-24T18:27:22.970 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.48223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43933.log 2024-04-24T18:27:22.970 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.62279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67832.log 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.62279.log.gz 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.43933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62877.log 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43933.log.gz 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67040.log 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61122.log 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62877.log: /var/log/ceph/cluster1-client.mirror.67040.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67832.log.gz 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.61122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53891.log 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62877.log.gz 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64860.log 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67040.log.gz 2024-04-24T18:27:22.971 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56980.log 2024-04-24T18:27:22.972 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70932.log 2024-04-24T18:27:22.972 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57803.log.gz 2024-04-24T18:27:22.972 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58436.log 2024-04-24T18:27:22.972 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46946.log 2024-04-24T18:27:22.972 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.58436.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64860.log.gz 2024-04-24T18:27:22.972 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51004.log 2024-04-24T18:27:22.972 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster1-client.admin.58436.log.gz 2024-04-24T18:27:22.972 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46946.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70932.log.gz 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45816.log 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59772.log.gz 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.51004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57398.log 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.45816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66146.log 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.57398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46364.log 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.66146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56828.log 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52327.log 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70911.log 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66146.log.gz 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54766.log 2024-04-24T18:27:22.975 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52327.log: /var/log/ceph/cluster1-client.mirror.70911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49114.log 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70038.log 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70911.log.gz 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.49114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49668.log 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67529.log 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59330.log 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70038.log.gz 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59990.log 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.59330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48655.log 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67529.log.gz 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62898.log 2024-04-24T18:27:22.978 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71265.log 2024-04-24T18:27:22.980 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70151.log 2024-04-24T18:27:22.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62898.log: /var/log/ceph/cluster1-client.mirror.71265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55819.log 2024-04-24T18:27:22.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55253.log 2024-04-24T18:27:22.980 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62898.log.gz 2024-04-24T18:27:22.980 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55819.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71265.log.gz 2024-04-24T18:27:22.980 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70151.log.gz 2024-04-24T18:27:22.981 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71977.log 2024-04-24T18:27:22.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.55253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56301.log 2024-04-24T18:27:22.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.71977.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71977.log.gz 2024-04-24T18:27:22.981 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59835.log 2024-04-24T18:27:22.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.56301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71286.log 2024-04-24T18:27:22.982 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-04-24T18:27:22.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54985.log 2024-04-24T18:27:22.983 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-osd.0.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58963.log.gz 2024-04-24T18:27:22.983 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58985.log.gz 2024-04-24T18:27:22.983 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71286.log.gz 2024-04-24T18:27:22.983 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57896.log 2024-04-24T18:27:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71024.log 2024-04-24T18:27:22.984 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57896.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59141.log.gz 2024-04-24T18:27:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71024.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60032.log.gz 2024-04-24T18:27:22.985 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71024.log.gz 2024-04-24T18:27:22.985 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46904.log 2024-04-24T18:27:22.986 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62679.log 2024-04-24T18:27:22.986 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45694.log 2024-04-24T18:27:22.986 INFO:teuthology.orchestra.run.smithi136.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.61310.log.gz 2024-04-24T18:27:22.987 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49329.log 2024-04-24T18:27:22.987 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.45694.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63386.log 2024-04-24T18:27:22.988 INFO:teuthology.orchestra.run.smithi136.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45985.log.gz 2024-04-24T18:27:22.988 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.49329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58509.log 2024-04-24T18:27:22.988 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62679.log.gz 2024-04-24T18:27:22.988 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.63386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71046.log 2024-04-24T18:27:22.988 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster2-client.admin.63386.log.gz 2024-04-24T18:27:22.988 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58509.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58177.log 2024-04-24T18:27:22.988 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51280.log 2024-04-24T18:27:22.988 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71046.log: /var/log/ceph/cluster2-client.mirror.58177.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50107.log 2024-04-24T18:27:22.990 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.51280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57022.log 2024-04-24T18:27:22.990 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.50107.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71046.log.gz 2024-04-24T18:27:22.991 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57697.log 2024-04-24T18:27:22.991 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.50234.log.gz 2024-04-24T18:27:22.991 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57022.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.35407.log 2024-04-24T18:27:22.991 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60493.log 2024-04-24T18:27:22.991 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71349.log 2024-04-24T18:27:22.991 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.tmp-client.admin.35407.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.35407.log.gz 2024-04-24T18:27:22.991 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71523.log 2024-04-24T18:27:22.991 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69567.log 2024-04-24T18:27:22.991 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.71523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61756.log 2024-04-24T18:27:22.992 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.71523.log.gz 2024-04-24T18:27:22.992 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72307.log 2024-04-24T18:27:22.992 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71349.log.gz 2024-04-24T18:27:22.992 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56807.log 2024-04-24T18:27:22.992 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52301.log 2024-04-24T18:27:22.992 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72307.log.gz 2024-04-24T18:27:22.992 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69567.log.gz 2024-04-24T18:27:22.993 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56807.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61756.log.gz 2024-04-24T18:27:22.994 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63046.log 2024-04-24T18:27:22.994 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.52301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45575.log 2024-04-24T18:27:22.994 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.63046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54639.log 2024-04-24T18:27:22.994 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.45575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50807.log 2024-04-24T18:27:22.994 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67557.log 2024-04-24T18:27:22.994 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54639.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63046.log.gz 2024-04-24T18:27:22.994 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50807.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66125.log 2024-04-24T18:27:22.996 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54724.log 2024-04-24T18:27:22.996 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.66125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44671.log 2024-04-24T18:27:22.996 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54724.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67557.log.gz 2024-04-24T18:27:22.996 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44671.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66125.log.gz 2024-04-24T18:27:22.996 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44671.log.gz 2024-04-24T18:27:22.996 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49248.log 2024-04-24T18:27:22.996 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50385.log 2024-04-24T18:27:22.998 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.49248.log: /var/log/ceph/cluster1-client.admin.50385.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52471.log.gz 2024-04-24T18:27:22.998 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.50385.log.gz 2024-04-24T18:27:22.998 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50587.log 2024-04-24T18:27:22.998 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55111.log 2024-04-24T18:27:22.998 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55798.log 2024-04-24T18:27:22.998 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.55111.log: /var/log/ceph/cluster2-client.mirror.55798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70509.log 2024-04-24T18:27:22.998 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62386.log 2024-04-24T18:27:22.999 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60538.log 2024-04-24T18:27:23.000 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52683.log.gz 2024-04-24T18:27:23.000 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62386.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52712.log.gz 2024-04-24T18:27:23.000 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70509.log.gz 2024-04-24T18:27:23.000 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67174.log 2024-04-24T18:27:23.000 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62386.log.gz 2024-04-24T18:27:23.000 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52838.log 2024-04-24T18:27:23.001 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.67174.log: /var/log/ceph/cluster1-client.mirror.52838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46925.log 2024-04-24T18:27:23.001 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54130.log 2024-04-24T18:27:23.001 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67174.log.gz 2024-04-24T18:27:23.001 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47452.log 2024-04-24T18:27:23.001 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.54130.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40662.log 2024-04-24T18:27:23.001 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59687.log 2024-04-24T18:27:23.003 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52274.log 2024-04-24T18:27:23.003 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.40662.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40662.log.gz 2024-04-24T18:27:23.003 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70653.log 2024-04-24T18:27:23.003 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.52274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66414.log 2024-04-24T18:27:23.003 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70653.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40550.log 2024-04-24T18:27:23.004 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.66414.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70653.log.gz 2024-04-24T18:27:23.004 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.40550.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40550.log.gz 2024-04-24T18:27:23.004 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54151.log 2024-04-24T18:27:23.004 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70017.log 2024-04-24T18:27:23.004 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.54151.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47891.log 2024-04-24T18:27:23.005 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70017.log: /var/log/ceph/cluster2-client.mirror.47891.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47388.log 2024-04-24T18:27:23.005 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67237.log 2024-04-24T18:27:23.005 INFO:teuthology.orchestra.run.smithi136.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46581.log.gz 2024-04-24T18:27:23.005 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70017.log.gz 2024-04-24T18:27:23.005 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46560.log.gz 2024-04-24T18:27:23.006 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47388.log: /var/log/ceph/cluster1-client.mirror.67237.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67237.log.gz 2024-04-24T18:27:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70826.log 2024-04-24T18:27:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49710.log 2024-04-24T18:27:23.007 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46707.log.gz 2024-04-24T18:27:23.007 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70826.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70826.log.gz 2024-04-24T18:27:23.009 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49710.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71550.log 2024-04-24T18:27:23.009 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68333.log 2024-04-24T18:27:23.009 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.71550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68248.log 2024-04-24T18:27:23.010 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68333.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.71550.log.gz 2024-04-24T18:27:23.010 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49994.log 2024-04-24T18:27:23.010 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51333.log 2024-04-24T18:27:23.011 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49994.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68248.log.gz 2024-04-24T18:27:23.011 INFO:teuthology.orchestra.run.smithi136.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68333.log.gz 2024-04-24T18:27:23.012 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58445.log 2024-04-24T18:27:23.015 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63282.log 2024-04-24T18:27:23.015 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.58445.log: /var/log/ceph/cluster1-client.admin.63282.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46946.log.gz 2024-04-24T18:27:23.015 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.51307.log.gz 2024-04-24T18:27:23.015 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.63282.log.gz 2024-04-24T18:27:23.015 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55231.log 2024-04-24T18:27:23.015 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57072.log 2024-04-24T18:27:23.015 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.55231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66468.log 2024-04-24T18:27:23.016 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71003.log 2024-04-24T18:27:23.017 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66468.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65822.log 2024-04-24T18:27:23.017 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55953.log 2024-04-24T18:27:23.017 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65822.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66468.log.gz 2024-04-24T18:27:23.017 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.55280.log.gz 2024-04-24T18:27:23.018 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71003.log.gz 2024-04-24T18:27:23.018 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65822.log.gz 2024-04-24T18:27:23.018 INFO:teuthology.orchestra.run.smithi136.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.55307.log.gz 2024-04-24T18:27:23.019 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66083.log 2024-04-24T18:27:23.019 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47693.log.gz 2024-04-24T18:27:23.019 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50310.log 2024-04-24T18:27:23.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.66083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54597.log 2024-04-24T18:27:23.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.50310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54210.log 2024-04-24T18:27:23.022 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58043.log 2024-04-24T18:27:23.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54597.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66083.log.gz 2024-04-24T18:27:23.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.54210.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51987.log 2024-04-24T18:27:23.022 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.54210.log.gz 2024-04-24T18:27:23.022 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.58043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69179.log 2024-04-24T18:27:23.022 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9%/var/log/ceph/cluster2-client.mirror.51987.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47891.log.gz 2024-04-24T18:27:23.022 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62743.log 2024-04-24T18:27:23.024 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66274.log 2024-04-24T18:27:23.024 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62743.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49001.log.gz 2024-04-24T18:27:23.024 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69179.log.gz 2024-04-24T18:27:23.024 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62743.log.gz 2024-04-24T18:27:23.024 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44901.log 2024-04-24T18:27:23.024 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.66274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57917.log 2024-04-24T18:27:23.024 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.44901.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.66274.log.gz 2024-04-24T18:27:23.025 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69116.log 2024-04-24T18:27:23.025 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44901.log.gz 2024-04-24T18:27:23.025 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59027.log 2024-04-24T18:27:23.025 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.48266.log.gz 2024-04-24T18:27:23.025 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44763.log 2024-04-24T18:27:23.025 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.59027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55465.log 2024-04-24T18:27:23.026 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59708.log 2024-04-24T18:27:23.026 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44763.log.gz 2024-04-24T18:27:23.026 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69116.log.gz 2024-04-24T18:27:23.026 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64600.log 2024-04-24T18:27:23.026 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54017.log 2024-04-24T18:27:23.027 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63960.log 2024-04-24T18:27:23.027 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.54017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71745.log 2024-04-24T18:27:23.027 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47954.log 2024-04-24T18:27:23.028 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64600.log.gz 2024-04-24T18:27:23.028 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.71745.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.71745.log.gz 2024-04-24T18:27:23.028 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57959.log 2024-04-24T18:27:23.028 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47954.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63960.log.gz 2024-04-24T18:27:23.028 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46447.log 2024-04-24T18:27:23.028 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49710.log.gz 2024-04-24T18:27:23.028 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60407.log 2024-04-24T18:27:23.029 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52093.log 2024-04-24T18:27:23.029 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.60407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63981.log 2024-04-24T18:27:23.029 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49794.log.gz 2024-04-24T18:27:23.030 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.52093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47870.log 2024-04-24T18:27:23.030 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67082.log 2024-04-24T18:27:23.030 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61078.log 2024-04-24T18:27:23.030 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70257.log 2024-04-24T18:27:23.031 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.67082.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63981.log.gz 2024-04-24T18:27:23.031 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.61078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55334.log 2024-04-24T18:27:23.031 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70257.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67082.log.gz 2024-04-24T18:27:23.031 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47474.log.gz 2024-04-24T18:27:23.031 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70257.log.gz 2024-04-24T18:27:23.032 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49213.log 2024-04-24T18:27:23.032 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.55334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52796.log 2024-04-24T18:27:23.032 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.49213.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/cluster1-client.admin.49213.log.gz-- 2024-04-24T18:27:23.032 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71109.log 2024-04-24T18:27:23.033 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52796.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46967.log.gz 2024-04-24T18:27:23.033 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58256.log 2024-04-24T18:27:23.033 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67216.log 2024-04-24T18:27:23.034 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.58256.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58256.log.gz 2024-04-24T18:27:23.034 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51591.log.gz 2024-04-24T18:27:23.034 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.67216.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67216.log.gz 2024-04-24T18:27:23.035 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67343.log 2024-04-24T18:27:23.035 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71109.log.gz 2024-04-24T18:27:23.035 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69996.log 2024-04-24T18:27:23.036 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.67343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45087.log 2024-04-24T18:27:23.036 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69996.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.67343.log.gz 2024-04-24T18:27:23.036 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51004.log.gz 2024-04-24T18:27:23.036 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51025.log.gz 2024-04-24T18:27:23.036 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53454.log.gz 2024-04-24T18:27:23.036 INFO:teuthology.orchestra.run.smithi136.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53196.log.gz 2024-04-24T18:27:23.036 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69996.log.gz 2024-04-24T18:27:23.037 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68051.log 2024-04-24T18:27:23.037 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.45087.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.45087.log.gz 2024-04-24T18:27:23.037 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61423.log 2024-04-24T18:27:23.037 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68051.log: /var/log/ceph/cluster2-client.admin.61423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64713.log 2024-04-24T18:27:23.038 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58942.log 2024-04-24T18:27:23.038 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68051.log.gz 2024-04-24T18:27:23.039 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57739.log 2024-04-24T18:27:23.039 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58942.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65688.log 2024-04-24T18:27:23.039 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8%/var/log/ceph/cluster2-client.mirror.57739.log: -- replaced with /var/log/ceph/cluster1-client.mirror.64713.log.gz 2024-04-24T18:27:23.040 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65688.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56079.log.gz 2024-04-24T18:27:23.040 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56100.log.gz 2024-04-24T18:27:23.040 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65688.log.gz 2024-04-24T18:27:23.040 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52662.log 2024-04-24T18:27:23.040 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53828.log 2024-04-24T18:27:23.041 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49355.log 2024-04-24T18:27:23.041 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53828.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55903.log 2024-04-24T18:27:23.041 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53240.log 2024-04-24T18:27:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48293.log 2024-04-24T18:27:23.042 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52838.log.gz 2024-04-24T18:27:23.042 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.53240.log: /var/log/ceph/cluster2-client.admin.48293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68959.log 2024-04-24T18:27:23.043 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58135.log 2024-04-24T18:27:23.044 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47933.log 2024-04-24T18:27:23.044 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.58135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45677.log 2024-04-24T18:27:23.044 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54787.log.gz 2024-04-24T18:27:23.044 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47933.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65949.log 2024-04-24T18:27:23.044 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68959.log.gz 2024-04-24T18:27:23.044 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54900.log.gz 2024-04-24T18:27:23.044 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54766.log.gz 2024-04-24T18:27:23.045 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.45677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66405.log 2024-04-24T18:27:23.045 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56651.log 2024-04-24T18:27:23.045 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.66405.log: 18.1% -- replaced with /var/log/ceph/cluster1-client.admin.66405.log.gz 2024-04-24T18:27:23.045 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56651.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53891.log.gz 2024-04-24T18:27:23.045 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65949.log.gz 2024-04-24T18:27:23.045 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65293.log 2024-04-24T18:27:23.046 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52247.log 2024-04-24T18:27:23.046 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.65293.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54985.log.gz 2024-04-24T18:27:23.046 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.52247.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.65293.log.gz 2024-04-24T18:27:23.046 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67474.log 2024-04-24T18:27:23.047 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55953.log.gz 2024-04-24T18:27:23.047 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50446.log 2024-04-24T18:27:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.67474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51443.log 2024-04-24T18:27:23.048 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster2-client.admin.67474.log.gz 2024-04-24T18:27:23.048 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.58332.log.gz 2024-04-24T18:27:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61245.log 2024-04-24T18:27:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51443.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63918.log 2024-04-24T18:27:23.048 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.61245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58849.log 2024-04-24T18:27:23.048 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.61245.log.gz 2024-04-24T18:27:23.048 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.59357.log.gz 2024-04-24T18:27:23.049 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60179.log 2024-04-24T18:27:23.049 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45884.log 2024-04-24T18:27:23.049 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56058.log.gz 2024-04-24T18:27:23.049 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63918.log.gz 2024-04-24T18:27:23.050 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.60179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50283.log 2024-04-24T18:27:23.050 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.45884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64818.log 2024-04-24T18:27:23.050 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.50283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66503.log 2024-04-24T18:27:23.051 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49135.log 2024-04-24T18:27:23.051 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68833.log 2024-04-24T18:27:23.051 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.49135.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64818.log.gz 2024-04-24T18:27:23.051 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68833.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48719.log.gz 2024-04-24T18:27:23.052 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66503.log.gz 2024-04-24T18:27:23.052 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56651.log.gz 2024-04-24T18:27:23.052 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48740.log.gz 2024-04-24T18:27:23.052 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68833.log.gz 2024-04-24T18:27:23.053 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68980.log 2024-04-24T18:27:23.053 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60880.log 2024-04-24T18:27:23.053 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72123.log 2024-04-24T18:27:23.054 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59383.log 2024-04-24T18:27:23.054 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72123.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72123.log.gz 2024-04-24T18:27:23.054 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68980.log.gz 2024-04-24T18:27:23.054 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53757.log 2024-04-24T18:27:23.054 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68329.log 2024-04-24T18:27:23.054 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50962.log 2024-04-24T18:27:23.055 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.68329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50941.log 2024-04-24T18:27:23.056 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.68329.log.gz 2024-04-24T18:27:23.056 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63939.log 2024-04-24T18:27:23.056 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52114.log 2024-04-24T18:27:23.056 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60158.log 2024-04-24T18:27:23.056 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.52114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49605.log 2024-04-24T18:27:23.057 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.60158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65928.log 2024-04-24T18:27:23.057 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63939.log.gz 2024-04-24T18:27:23.057 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49605.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51811.log.gz 2024-04-24T18:27:23.059 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49584.log.gz 2024-04-24T18:27:23.059 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54963.log 2024-04-24T18:27:23.060 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68135.log 2024-04-24T18:27:23.060 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53673.log 2024-04-24T18:27:23.061 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61890.log 2024-04-24T18:27:23.061 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65928.log.gz 2024-04-24T18:27:23.061 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53673.log: /var/log/ceph/cluster2-client.mirror.61890.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68135.log.gz 2024-04-24T18:27:23.062 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61890.log.gz 2024-04-24T18:27:23.062 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65730.log 2024-04-24T18:27:23.063 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60116.log 2024-04-24T18:27:23.063 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43935.log 2024-04-24T18:27:23.063 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.60116.log: /var/log/ceph/cluster2-client.admin.43935.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43935.log.gz 2024-04-24T18:27:23.063 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65730.log.gz 2024-04-24T18:27:23.064 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63634.log 2024-04-24T18:27:23.064 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63634.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57917.log.gz 2024-04-24T18:27:23.064 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63634.log.gz 2024-04-24T18:27:23.065 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60241.log 2024-04-24T18:27:23.066 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49217.log 2024-04-24T18:27:23.066 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.60241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45711.log 2024-04-24T18:27:23.066 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.60241.log.gz 2024-04-24T18:27:23.066 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.49217.log: /var/log/ceph/cluster2-client.admin.45711.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.45711.log.gz 2024-04-24T18:27:23.066 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60923.log 2024-04-24T18:27:23.067 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59162.log 2024-04-24T18:27:23.067 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60923.log: /var/log/ceph/cluster1-client.mirror.59162.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58156.log.gz 2024-04-24T18:27:23.067 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60398.log 2024-04-24T18:27:23.067 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58135.log.gz 2024-04-24T18:27:23.067 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46186.log 2024-04-24T18:27:23.068 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.60398.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56226.log 2024-04-24T18:27:23.068 INFO:teuthology.orchestra.run.smithi136.stderr: 17.5% -- replaced with /var/log/ceph/cluster1-client.admin.60398.log.gz 2024-04-24T18:27:23.068 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46186.log: /var/log/ceph/cluster2-client.mirror.49434.log 2024-04-24T18:27:23.069 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.56226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67790.log 2024-04-24T18:27:23.069 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47778.log 2024-04-24T18:27:23.070 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44073.log 2024-04-24T18:27:23.070 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67790.log: /var/log/ceph/cluster2-client.mirror.47778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63783.log 2024-04-24T18:27:23.070 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44073.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44073.log.gz 2024-04-24T18:27:23.070 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67790.log.gz 2024-04-24T18:27:23.070 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50015.log 2024-04-24T18:27:23.070 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63783.log.gz 2024-04-24T18:27:23.071 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65886.log 2024-04-24T18:27:23.071 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.50015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70531.log 2024-04-24T18:27:23.072 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60724.log 2024-04-24T18:27:23.072 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.70531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47495.log 2024-04-24T18:27:23.072 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70531.log.gz 2024-04-24T18:27:23.072 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65426.log 2024-04-24T18:27:23.072 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60724.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65886.log.gz 2024-04-24T18:27:23.072 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47996.log 2024-04-24T18:27:23.074 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.65426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61015.log 2024-04-24T18:27:23.075 INFO:teuthology.orchestra.run.smithi136.stderr: 18.7% -- replaced with /var/log/ceph/cluster2-client.admin.65426.log.gz 2024-04-24T18:27:23.075 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69862.log 2024-04-24T18:27:23.075 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.61015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51366.log 2024-04-24T18:27:23.075 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62962.log 2024-04-24T18:27:23.075 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.51366.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55692.log 2024-04-24T18:27:23.075 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53321.log 2024-04-24T18:27:23.075 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69862.log.gz 2024-04-24T18:27:23.075 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71625.log 2024-04-24T18:27:23.076 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.53321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62722.log 2024-04-24T18:27:23.076 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62962.log.gz 2024-04-24T18:27:23.076 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.71625.log: 0.0%gzip -5 -- replaced with /var/log/ceph/cluster2-client.admin.71625.log.gz--verbose 2024-04-24T18:27:23.076 INFO:teuthology.orchestra.run.smithi136.stderr:-- /var/log/ceph/cluster2-client.mirror.61735.log 2024-04-24T18:27:23.076 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67472.log 2024-04-24T18:27:23.077 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69242.log 2024-04-24T18:27:23.077 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.67472.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62722.log.gz 2024-04-24T18:27:23.077 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.67472.log.gz 2024-04-24T18:27:23.077 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69242.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61735.log.gz 2024-04-24T18:27:23.078 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55663.log.gz 2024-04-24T18:27:23.078 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67308.log 2024-04-24T18:27:23.078 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69242.log.gz 2024-04-24T18:27:23.078 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70215.log 2024-04-24T18:27:23.078 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59027.log.gz 2024-04-24T18:27:23.079 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.67308.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67308.log.gz 2024-04-24T18:27:23.079 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66956.log 2024-04-24T18:27:23.079 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54829.log 2024-04-24T18:27:23.079 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66956.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63868.log 2024-04-24T18:27:23.079 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70215.log.gz 2024-04-24T18:27:23.080 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54829.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66956.log.gz 2024-04-24T18:27:23.081 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63868.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63868.log.gz 2024-04-24T18:27:23.081 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49752.log 2024-04-24T18:27:23.081 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58849.log.gz 2024-04-24T18:27:23.082 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58879.log.gz 2024-04-24T18:27:23.082 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54858.log 2024-04-24T18:27:23.082 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58942.log.gz 2024-04-24T18:27:23.082 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51916.log 2024-04-24T18:27:23.082 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54858.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51612.log 2024-04-24T18:27:23.084 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55498.log 2024-04-24T18:27:23.084 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51253.log 2024-04-24T18:27:23.084 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72215.log 2024-04-24T18:27:23.084 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.51253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47224.log 2024-04-24T18:27:23.084 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54745.log 2024-04-24T18:27:23.084 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72215.log.gz 2024-04-24T18:27:23.085 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.47224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47516.log 2024-04-24T18:27:23.085 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54745.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50555.log.gz 2024-04-24T18:27:23.085 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50587.log.gz 2024-04-24T18:27:23.087 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60703.log 2024-04-24T18:27:23.087 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67874.log 2024-04-24T18:27:23.087 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50396.log 2024-04-24T18:27:23.088 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62360.log 2024-04-24T18:27:23.088 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.50396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52880.log 2024-04-24T18:27:23.088 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.62360.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67874.log.gz 2024-04-24T18:27:23.088 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.62360.log.gz 2024-04-24T18:27:23.089 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54109.log.gz 2024-04-24T18:27:23.091 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58434.log 2024-04-24T18:27:23.091 INFO:teuthology.orchestra.run.smithi136.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54214.log.gz 2024-04-24T18:27:23.091 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54376.log 2024-04-24T18:27:23.091 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.58434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58434.log.gz 2024-04-24T18:27:23.092 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47111.log 2024-04-24T18:27:23.092 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.54376.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54551.log.gz 2024-04-24T18:27:23.092 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54509.log.gz 2024-04-24T18:27:23.093 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57270.log 2024-04-24T18:27:23.093 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.47111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57875.log 2024-04-24T18:27:23.093 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64210.log 2024-04-24T18:27:23.093 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55776.log 2024-04-24T18:27:23.093 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.64210.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58305.log 2024-04-24T18:27:23.093 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64210.log.gz 2024-04-24T18:27:23.093 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63067.log 2024-04-24T18:27:23.094 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.58305.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68156.log 2024-04-24T18:27:23.094 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.63067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48698.log 2024-04-24T18:27:23.096 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68156.log: /var/log/ceph/cluster1-client.mirror.48698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67195.log 2024-04-24T18:27:23.096 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63067.log.gz 2024-04-24T18:27:23.096 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50722.log 2024-04-24T18:27:23.096 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68156.log.gz 2024-04-24T18:27:23.096 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.67195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54703.log 2024-04-24T18:27:23.096 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69513.log 2024-04-24T18:27:23.096 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54703.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67195.log.gz 2024-04-24T18:27:23.097 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.69513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63004.log 2024-04-24T18:27:23.097 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67312.log 2024-04-24T18:27:23.098 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.63004.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57043.log.gz 2024-04-24T18:27:23.098 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63004.log.gz 2024-04-24T18:27:23.099 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57189.log.gz 2024-04-24T18:27:23.101 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44855.log 2024-04-24T18:27:23.101 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.67312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70194.log 2024-04-24T18:27:23.101 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.44855.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44855.log.gz 2024-04-24T18:27:23.102 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48790.log 2024-04-24T18:27:23.102 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70194.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67312.log.gz 2024-04-24T18:27:23.103 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49093.log 2024-04-24T18:27:23.103 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70194.log.gz 2024-04-24T18:27:23.103 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67938.log 2024-04-24T18:27:23.104 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.49093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44809.log 2024-04-24T18:27:23.104 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67938.log: /var/log/ceph/cluster1-client.admin.44809.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44809.log.gz 2024-04-24T18:27:23.104 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70805.log 2024-04-24T18:27:23.104 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67938.log.gz 2024-04-24T18:27:23.105 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56895.log 2024-04-24T18:27:23.105 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70805.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70805.log.gz 2024-04-24T18:27:23.107 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48853.log 2024-04-24T18:27:23.107 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46539.log.gz 89.9% 2024-04-24T18:27:23.107 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.46728.log.gz 2024-04-24T18:27:23.107 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46757.log.gz 2024-04-24T18:27:23.108 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59637.log 2024-04-24T18:27:23.108 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43596.log 2024-04-24T18:27:23.109 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51966.log 2024-04-24T18:27:23.109 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.51280.log.gz 2024-04-24T18:27:23.109 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.43596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64839.log 2024-04-24T18:27:23.109 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43596.log.gz 2024-04-24T18:27:23.109 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46904.log.gz 2024-04-24T18:27:23.110 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64776.log 2024-04-24T18:27:23.110 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68570.log 2024-04-24T18:27:23.110 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64776.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.49329.log.gz 2024-04-24T18:27:23.111 INFO:teuthology.orchestra.run.smithi136.stderr: 18.7% -- replaced with /var/log/ceph/cluster1-client.admin.46186.log.gz 2024-04-24T18:27:23.111 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64839.log.gz 2024-04-24T18:27:23.111 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64776.log.gz 2024-04-24T18:27:23.114 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59535.log 2024-04-24T18:27:23.115 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68570.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68570.log.gz 2024-04-24T18:27:23.116 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58114.log 2024-04-24T18:27:23.116 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59535.log: /var/log/ceph/cluster2-client.mirror.58114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48761.log 2024-04-24T18:27:23.117 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46447.log.gz 2024-04-24T18:27:23.117 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57473.log 2024-04-24T18:27:23.117 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46292.log 2024-04-24T18:27:23.117 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.57473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66252.log 2024-04-24T18:27:23.118 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.57473.log.gz 2024-04-24T18:27:23.118 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.52247.log.gz 2024-04-24T18:27:23.123 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46292.log: /var/log/ceph/cluster2-client.mirror.66252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57210.log 2024-04-24T18:27:23.123 INFO:teuthology.orchestra.run.smithi136.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66252.log.gz 2024-04-24T18:27:23.123 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63088.log 2024-04-24T18:27:23.123 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57210.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48790.log.gz 2024-04-24T18:27:23.123 INFO:teuthology.orchestra.run.smithi136.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.45528.log.gz 2024-04-24T18:27:23.124 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.50283.log.gz 2024-04-24T18:27:23.124 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.50310.log.gz 2024-04-24T18:27:23.124 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48761.log.gz 2024-04-24T18:27:23.124 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64235.log 2024-04-24T18:27:23.125 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.63088.log: 87.3%/var/log/ceph/cluster2-client.admin.64235.log: -- replaced with /var/log/ceph/cluster1-client.admin.47111.log.gz 2024-04-24T18:27:23.125 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46318.log.gz 2024-04-24T18:27:23.125 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63088.log.gz 2024-04-24T18:27:23.125 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58900.log 2024-04-24T18:27:23.125 INFO:teuthology.orchestra.run.smithi136.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.64235.log.gz 2024-04-24T18:27:23.125 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44625.log 2024-04-24T18:27:23.126 INFO:teuthology.orchestra.run.smithi136.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45694.log.gz 2024-04-24T18:27:23.126 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66977.log 2024-04-24T18:27:23.126 INFO:teuthology.orchestra.run.smithi136.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.45765.log.gz 2024-04-24T18:27:23.126 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.44625.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.45782.log.gz 2024-04-24T18:27:23.126 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44625.log.gz 2024-04-24T18:27:23.126 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66033.log 2024-04-24T18:27:23.127 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66977.log: /var/log/ceph/cluster2-client.mirror.66033.log: 54.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45816.log.gz 2024-04-24T18:27:23.127 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66977.log.gz 2024-04-24T18:27:23.127 INFO:teuthology.orchestra.run.smithi136.stderr: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45833.log.gz 2024-04-24T18:27:23.127 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52620.log 2024-04-24T18:27:23.128 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66033.log.gz 2024-04-24T18:27:23.128 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62172.log 2024-04-24T18:27:23.128 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59303.log 2024-04-24T18:27:23.128 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.62172.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49605.log.gz 2024-04-24T18:27:23.129 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68642.log 2024-04-24T18:27:23.129 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.59303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67397.log 2024-04-24T18:27:23.129 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62172.log.gz 2024-04-24T18:27:23.130 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68642.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47933.log.gz 2024-04-24T18:27:23.131 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68642.log.gz 2024-04-24T18:27:23.131 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47954.log.gz 2024-04-24T18:27:23.131 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54365.log 2024-04-24T18:27:23.131 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.67397.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48320.log 2024-04-24T18:27:23.131 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51443.log.gz 2024-04-24T18:27:23.131 INFO:teuthology.orchestra.run.smithi136.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.67397.log.gz 2024-04-24T18:27:23.132 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.54365.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54365.log.gz 2024-04-24T18:27:23.132 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69594.log 2024-04-24T18:27:23.132 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.48320.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49093.log.gz 2024-04-24T18:27:23.133 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69594.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49135.log.gz 2024-04-24T18:27:23.133 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69594.log.gz 2024-04-24T18:27:23.133 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67988.log 2024-04-24T18:27:23.133 INFO:teuthology.orchestra.run.smithi136.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49217.log.gz 2024-04-24T18:27:23.133 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59006.log 2024-04-24T18:27:23.134 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63160.log 2024-04-24T18:27:23.134 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.59006.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52620.log.gz 2024-04-24T18:27:23.134 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.63160.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67988.log.gz 2024-04-24T18:27:23.135 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63160.log.gz 2024-04-24T18:27:23.135 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65843.log 2024-04-24T18:27:23.135 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52051.log.gz 2024-04-24T18:27:23.137 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65843.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65843.log.gz 2024-04-24T18:27:23.137 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66355.log 2024-04-24T18:27:23.138 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.66355.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52199.log.gz 2024-04-24T18:27:23.138 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53828.log.gz 2024-04-24T18:27:23.138 INFO:teuthology.orchestra.run.smithi136.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.66355.log.gz 2024-04-24T18:27:23.138 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43659.log 2024-04-24T18:27:23.138 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52754.log 2024-04-24T18:27:23.139 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.43659.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43659.log.gz 2024-04-24T18:27:23.139 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45918.log 2024-04-24T18:27:23.140 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59948.log 2024-04-24T18:27:23.140 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54858.log.gz 2024-04-24T18:27:23.140 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.45918.log: /var/log/ceph/cluster2-client.mirror.59948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52972.log 2024-04-24T18:27:23.140 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54829.log.gz 2024-04-24T18:27:23.140 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64994.log 2024-04-24T18:27:23.150 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52972.log: /var/log/ceph/cluster1-client.mirror.64994.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53757.log.gz 2024-04-24T18:27:23.150 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54963.log.gz 2024-04-24T18:27:23.150 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64994.log.gz 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55384.log 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50983.log 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.55384.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56477.log.gz 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50446.log.gz 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45974.log 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69297.log 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59183.log 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.45974.log: /var/log/ceph/cluster1-client.admin.69297.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.69297.log.gz 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46799.log 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.59183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50679.log 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.54236.log 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64065.log 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50679.log: /var/log/ceph/cluster2-client.admin.54236.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55776.log.gz 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57210.log.gz 2024-04-24T18:27:23.151 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62151.log 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.64065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44995.log 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.62151.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64065.log.gz 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62151.log.gz 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45503.log 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60640.log.gz 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.44995.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44995.log.gz 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.45503.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60538.log.gz 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45503.log.gz 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66328.log 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52351.log 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.66328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61693.log 2024-04-24T18:27:23.152 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.66328.log.gz 2024-04-24T18:27:23.153 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.52351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69502.log 2024-04-24T18:27:23.153 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61693.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60859.log.gz 2024-04-24T18:27:23.153 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49275.log 2024-04-24T18:27:23.153 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61693.log.gz 2024-04-24T18:27:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.69502.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60880.log.gz 2024-04-24T18:27:23.154 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69502.log.gz 2024-04-24T18:27:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47651.log 2024-04-24T18:27:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.49275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47027.log 2024-04-24T18:27:23.154 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68430.log 2024-04-24T18:27:23.155 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47651.log: /var/log/ceph/cluster1-client.mirror.47027.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52360.log 2024-04-24T18:27:23.155 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.68430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52599.log 2024-04-24T18:27:23.155 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.68430.log.gz 2024-04-24T18:27:23.155 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.52360.log: /var/log/ceph/cluster1-client.mirror.52599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68219.log 2024-04-24T18:27:23.155 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65516.log 2024-04-24T18:27:23.156 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58900.log.gz 2024-04-24T18:27:23.156 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68219.log: /var/log/ceph/cluster2-client.mirror.65516.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59006.log.gz 2024-04-24T18:27:23.156 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65516.log.gz 2024-04-24T18:27:23.156 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51088.log 2024-04-24T18:27:23.157 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68219.log.gz 2024-04-24T18:27:23.157 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64383.log 2024-04-24T18:27:23.158 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.51088.log: /var/log/ceph/cluster1-client.admin.64383.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.51226.log.gz 2024-04-24T18:27:23.158 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56354.log.gz 2024-04-24T18:27:23.158 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46820.log 2024-04-24T18:27:23.159 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66104.log 2024-04-24T18:27:23.159 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster2-client.admin.55384.log.gz 2024-04-24T18:27:23.160 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46820.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59183.log.gz 2024-04-24T18:27:23.161 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.66104.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66403.log 2024-04-24T18:27:23.162 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59535.log.gz 2024-04-24T18:27:23.162 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66104.log.gz 2024-04-24T18:27:23.162 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71307.log 2024-04-24T18:27:23.162 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56895.log.gz 2024-04-24T18:27:23.163 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.66403.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66403.log.gz 2024-04-24T18:27:23.163 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45901.log 2024-04-24T18:27:23.163 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59637.log.gz 2024-04-24T18:27:23.164 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46497.log.gz 2024-04-24T18:27:23.164 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.45901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64642.log 2024-04-24T18:27:23.165 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46799.log.gz 2024-04-24T18:27:23.165 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46820.log.gz 2024-04-24T18:27:23.165 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71307.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71307.log.gz 2024-04-24T18:27:23.167 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65709.log 2024-04-24T18:27:23.167 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64642.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64642.log.gz 2024-04-24T18:27:23.169 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65625.log 2024-04-24T18:27:23.169 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65709.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65709.log.gz 2024-04-24T18:27:23.170 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61827.log 2024-04-24T18:27:23.170 INFO:teuthology.orchestra.run.smithi136.stderr: 18.7% -- replaced with /var/log/ceph/cluster1-client.admin.52351.log.gz 2024-04-24T18:27:23.171 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61827.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65625.log.gz 2024-04-24T18:27:23.172 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70173.log 2024-04-24T18:27:23.172 INFO:teuthology.orchestra.run.smithi136.stderr: 67.2% -- replaced with /var/log/ceph/cluster1-client.admin.45575.log.gz 2024-04-24T18:27:23.172 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61827.log.gz 2024-04-24T18:27:23.173 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70173.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70173.log.gz 2024-04-24T18:27:23.173 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50743.log 2024-04-24T18:27:23.174 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65646.log 2024-04-24T18:27:23.174 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49355.log.gz 2024-04-24T18:27:23.175 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50743.log: /var/log/ceph/cluster2-client.mirror.65646.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65646.log.gz 2024-04-24T18:27:23.175 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56121.log 2024-04-24T18:27:23.175 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52796.log.gz 2024-04-24T18:27:23.176 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.56121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61672.log 2024-04-24T18:27:23.176 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50616.log.gz 2024-04-24T18:27:23.181 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61672.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61672.log.gz 2024-04-24T18:27:23.181 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64086.log 2024-04-24T18:27:23.181 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52880.log.gz 2024-04-24T18:27:23.181 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50679.log.gz 2024-04-24T18:27:23.181 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.64086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62637.log 2024-04-24T18:27:23.181 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62637.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64086.log.gz 2024-04-24T18:27:23.182 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.59303.log.gz 2024-04-24T18:27:23.182 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62637.log.gz 2024-04-24T18:27:23.182 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65991.log 2024-04-24T18:27:23.182 INFO:teuthology.orchestra.run.smithi136.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48223.log.gz 2024-04-24T18:27:23.183 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47312.log.gz 2024-04-24T18:27:23.184 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65991.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51088.log.gz 2024-04-24T18:27:23.185 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65991.log.gz 2024-04-24T18:27:23.185 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49952.log 2024-04-24T18:27:23.185 INFO:teuthology.orchestra.run.smithi136.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.52301.log.gz 2024-04-24T18:27:23.185 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47651.log.gz 2024-04-24T18:27:23.186 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-04-24T18:27:23.187 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51416.log.gz 2024-04-24T18:27:23.187 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68619.log 2024-04-24T18:27:23.188 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68619.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52515.log.gz 2024-04-24T18:27:23.189 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68619.log.gz 2024-04-24T18:27:23.189 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67895.log 2024-04-24T18:27:23.189 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62016.log 2024-04-24T18:27:23.190 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67895.log: /var/log/ceph/cluster2-client.mirror.62016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67061.log 2024-04-24T18:27:23.191 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67895.log.gz 2024-04-24T18:27:23.191 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.67061.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62016.log.gz 2024-04-24T18:27:23.191 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51706.log 2024-04-24T18:27:23.191 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66778.log 2024-04-24T18:27:23.192 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67061.log.gz 2024-04-24T18:27:23.192 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51706.log: /var/log/ceph/cluster1-client.mirror.66778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47975.log 2024-04-24T18:27:23.192 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.49248.log.gz 2024-04-24T18:27:23.192 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67581.log 2024-04-24T18:27:23.193 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47975.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66778.log.gz 2024-04-24T18:27:23.193 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52557.log.gz 2024-04-24T18:27:23.193 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68938.log 2024-04-24T18:27:23.195 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67581.log: /var/log/ceph/cluster1-client.mirror.68938.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52599.log.gz 2024-04-24T18:27:23.195 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67581.log.gz 2024-04-24T18:27:23.195 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51748.log 2024-04-24T18:27:23.195 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68938.log.gz 2024-04-24T18:27:23.195 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61869.log 2024-04-24T18:27:23.196 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60321.log 2024-04-24T18:27:23.196 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61869.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61869.log.gz 2024-04-24T18:27:23.197 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48520.log 2024-04-24T18:27:23.198 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.60321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49866.log 2024-04-24T18:27:23.201 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62526.log 2024-04-24T18:27:23.201 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49866.log: /var/log/ceph/cluster1-client.mirror.62526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48980.log 2024-04-24T18:27:23.202 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48980.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45918.log.gz 2024-04-24T18:27:23.202 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52072.log 2024-04-24T18:27:23.202 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62526.log.gz 2024-04-24T18:27:23.203 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.52072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58638.log 2024-04-24T18:27:23.203 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52754.log.gz 2024-04-24T18:27:23.204 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52972.log.gz 2024-04-24T18:27:23.204 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55027.log.gz 2024-04-24T18:27:23.205 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61101.log 2024-04-24T18:27:23.206 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58638.log: /var/log/ceph/cluster1-client.mirror.61101.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59919.log 2024-04-24T18:27:23.206 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57072.log.gz 2024-04-24T18:27:23.206 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56121.log.gz 2024-04-24T18:27:23.207 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64157.log 2024-04-24T18:27:23.207 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58539.log 2024-04-24T18:27:23.208 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.64157.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64157.log.gz 2024-04-24T18:27:23.208 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59948.log.gz 2024-04-24T18:27:23.209 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58786.log 2024-04-24T18:27:23.209 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61364.log 2024-04-24T18:27:23.209 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71720.log 2024-04-24T18:27:23.211 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.61364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61187.log 2024-04-24T18:27:23.211 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.71720.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71720.log.gz 2024-04-24T18:27:23.211 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59383.log.gz 2024-04-24T18:27:23.212 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50743.log.gz 2024-04-24T18:27:23.212 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59969.log.gz 2024-04-24T18:27:23.214 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.61187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46426.log 2024-04-24T18:27:23.214 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49377.log 2024-04-24T18:27:23.215 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46426.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53715.log.gz 2024-04-24T18:27:23.215 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.49377.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49377.log.gz 2024-04-24T18:27:23.216 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50387.log 2024-04-24T18:27:23.216 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67124.log 2024-04-24T18:27:23.217 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.50387.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.54290.log.gz 2024-04-24T18:27:23.218 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52641.log 2024-04-24T18:27:23.219 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.67124.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67124.log.gz 2024-04-24T18:27:23.219 INFO:teuthology.orchestra.run.smithi136.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45901.log.gz 2024-04-24T18:27:23.219 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59814.log 2024-04-24T18:27:23.220 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52922.log 2024-04-24T18:27:23.220 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.53240.log.gz 2024-04-24T18:27:23.221 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46426.log.gz 2024-04-24T18:27:23.221 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.52274.log.gz 2024-04-24T18:27:23.221 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53954.log 2024-04-24T18:27:23.221 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52922.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46468.log.gz 2024-04-24T18:27:23.222 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63468.log 2024-04-24T18:27:23.223 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56378.log 2024-04-24T18:27:23.225 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63468.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63468.log.gz 2024-04-24T18:27:23.225 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.54263.log.gz 2024-04-24T18:27:23.225 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70299.log 2024-04-24T18:27:23.226 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.56378.log: /var/log/ceph/cluster2-client.mirror.70299.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50983.log.gz 2024-04-24T18:27:23.226 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70299.log.gz 2024-04-24T18:27:23.226 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63804.log 2024-04-24T18:27:23.227 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67019.log 2024-04-24T18:27:23.227 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63804.log: /var/log/ceph/cluster1-client.mirror.67019.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63804.log.gz 2024-04-24T18:27:23.228 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67019.log.gz 2024-04-24T18:27:23.228 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63490.log 2024-04-24T18:27:23.228 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54879.log 2024-04-24T18:27:23.229 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63490.log: /var/log/ceph/cluster1-client.mirror.54879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57371.log 2024-04-24T18:27:23.230 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63490.log.gz 2024-04-24T18:27:23.230 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47912.log.gz 2024-04-24T18:27:23.230 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47975.log.gz 2024-04-24T18:27:23.231 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53912.log 2024-04-24T18:27:23.231 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.57371.log: /var/log/ceph/cluster2-client.mirror.53912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56917.log 2024-04-24T18:27:23.232 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72581.log 2024-04-24T18:27:23.232 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56917.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48520.log.gz 2024-04-24T18:27:23.233 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69093.log 2024-04-24T18:27:23.233 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72581.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72581.log.gz 2024-04-24T18:27:23.234 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54639.log.gz 2024-04-24T18:27:23.236 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61283.log 2024-04-24T18:27:23.236 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69093.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69093.log.gz 2024-04-24T18:27:23.236 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62059.log 2024-04-24T18:27:23.238 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.61283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64734.log 2024-04-24T18:27:23.238 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.62059.log: /var/log/ceph/cluster1-client.mirror.64734.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62059.log.gz 2024-04-24T18:27:23.238 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71496.log 2024-04-24T18:27:23.238 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64734.log.gz 2024-04-24T18:27:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62468.log 2024-04-24T18:27:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.71496.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.71496.log.gz 2024-04-24T18:27:23.239 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55995.log 2024-04-24T18:27:23.240 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62468.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62468.log.gz 2024-04-24T18:27:23.240 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65490.log 2024-04-24T18:27:23.241 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55995.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67698.log 2024-04-24T18:27:23.242 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68739.log 2024-04-24T18:27:23.242 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67698.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65490.log.gz 2024-04-24T18:27:23.242 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67698.log.gz 2024-04-24T18:27:23.243 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56851.log 2024-04-24T18:27:23.244 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68739.log: /var/log/ceph/cluster1-client.mirror.56851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48373.log 2024-04-24T18:27:23.244 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68739.log.gz 2024-04-24T18:27:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58561.log 2024-04-24T18:27:23.245 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50807.log.gz 2024-04-24T18:27:23.245 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70456.log 2024-04-24T18:27:23.246 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68507.log 2024-04-24T18:27:23.246 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.70456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57655.log 2024-04-24T18:27:23.247 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.68507.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.70456.log.gz 2024-04-24T18:27:23.247 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster1-client.admin.68507.log.gz 2024-04-24T18:27:23.247 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59687.log.gz 2024-04-24T18:27:23.247 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.60321.log.gz 2024-04-24T18:27:23.249 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51706.log.gz 2024-04-24T18:27:23.250 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52072.log.gz 2024-04-24T18:27:23.250 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46292.log.gz 2024-04-24T18:27:23.252 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52641.log.gz 2024-04-24T18:27:23.252 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69504.log 2024-04-24T18:27:23.252 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52922.log.gz 2024-04-24T18:27:23.252 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70542.log 2024-04-24T18:27:23.252 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.69504.log: 18.1% -- replaced with /var/log/ceph/cluster2-client.admin.69504.log.gz 2024-04-24T18:27:23.252 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53954.log.gz 2024-04-24T18:27:23.253 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54151.log.gz 2024-04-24T18:27:23.253 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54724.log.gz 2024-04-24T18:27:23.253 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54879.log.gz 2024-04-24T18:27:23.254 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55798.log.gz 2024-04-24T18:27:23.254 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.61283.log.gz 2024-04-24T18:27:23.261 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.70542.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55995.log.gz 2024-04-24T18:27:23.262 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72491.log 2024-04-24T18:27:23.262 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60514.log 2024-04-24T18:27:23.262 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72491.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72491.log.gz 2024-04-24T18:27:23.262 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46195.log 2024-04-24T18:27:23.266 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.46195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62806.log 2024-04-24T18:27:23.266 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62806.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62806.log.gz 2024-04-24T18:27:23.266 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57344.log 2024-04-24T18:27:23.266 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51748.log.gz 2024-04-24T18:27:23.266 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51832.log.gz 2024-04-24T18:27:23.268 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68896.log 2024-04-24T18:27:23.268 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.57344.log: /var/log/ceph/cluster1-client.mirror.68896.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68896.log.gz 2024-04-24T18:27:23.268 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65347.log 2024-04-24T18:27:23.269 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53778.log.gz 2024-04-24T18:27:23.269 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57655.log.gz 2024-04-24T18:27:23.269 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57634.log.gz 2024-04-24T18:27:23.270 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.65347.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.65347.log.gz 2024-04-24T18:27:23.270 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45643.log 2024-04-24T18:27:23.271 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58638.log.gz 2024-04-24T18:27:23.272 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.45643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61503.log 2024-04-24T18:27:23.272 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63139.log 2024-04-24T18:27:23.273 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61503.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61503.log.gz 2024-04-24T18:27:23.273 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48980.log.gz 2024-04-24T18:27:23.275 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.63139.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63139.log.gz 2024-04-24T18:27:23.275 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50899.log 2024-04-24T18:27:23.276 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48811.log 2024-04-24T18:27:23.277 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50899.log: /var/log/ceph/cluster1-client.mirror.48811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61337.log 2024-04-24T18:27:23.277 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58764.log 2024-04-24T18:27:23.278 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.61337.log: /var/log/ceph/cluster1-client.mirror.58764.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60923.log.gz 2024-04-24T18:27:23.279 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60095.log 2024-04-24T18:27:23.279 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60245.log 2024-04-24T18:27:23.281 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.60095.log: /var/log/ceph/cluster2-client.mirror.60245.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49994.log.gz 2024-04-24T18:27:23.281 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49952.log.gz 2024-04-24T18:27:23.281 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47023.log 2024-04-24T18:27:23.282 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.47023.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.47023.log.gz 2024-04-24T18:27:23.282 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.51357.log 2024-04-24T18:27:23.282 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.51357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46686.log 2024-04-24T18:27:23.283 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67153.log 2024-04-24T18:27:23.284 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.67153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51570.log 2024-04-24T18:27:23.284 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67153.log.gz 2024-04-24T18:27:23.285 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50036.log 2024-04-24T18:27:23.286 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53610.log 2024-04-24T18:27:23.289 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.50036.log: /var/log/ceph/cluster2-client.mirror.53610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48406.log 2024-04-24T18:27:23.289 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.48406.log: 18.1% -- replaced with /var/log/ceph/cluster1-client.admin.56378.log.gz 2024-04-24T18:27:23.290 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50764.log 2024-04-24T18:27:23.290 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52993.log 2024-04-24T18:27:23.291 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46341.log.gz 2024-04-24T18:27:23.291 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.51253.log.gz 2024-04-24T18:27:23.291 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68269.log 2024-04-24T18:27:23.292 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.70377.log 2024-04-24T18:27:23.293 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68269.log: /var/log/ceph/cluster2-client.admin.70377.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.70377.log.gz 2024-04-24T18:27:23.293 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45609.log 2024-04-24T18:27:23.294 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68269.log.gz 2024-04-24T18:27:23.295 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.45609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63847.log 2024-04-24T18:27:23.297 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49908.log 2024-04-24T18:27:23.297 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63847.log.gz 2024-04-24T18:27:23.297 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47516.log.gz 2024-04-24T18:27:23.299 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48373.log.gz 2024-04-24T18:27:23.300 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47495.log.gz 2024-04-24T18:27:23.302 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.59330.log.gz 2024-04-24T18:27:23.302 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47996.log.gz 2024-04-24T18:27:23.303 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50962.log.gz 2024-04-24T18:27:23.304 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50764.log.gz 2024-04-24T18:27:23.304 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52114.log.gz 2024-04-24T18:27:23.304 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55621.log.gz 2024-04-24T18:27:23.307 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52093.log.gz 2024-04-24T18:27:23.307 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52993.log.gz 2024-04-24T18:27:23.307 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51570.log.gz 2024-04-24T18:27:23.307 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53610.log.gz 2024-04-24T18:27:23.307 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49773.log.gz 2024-04-24T18:27:23.307 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52420.log.gz 2024-04-24T18:27:23.307 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51684.log.gz 2024-04-24T18:27:23.307 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53849.log.gz 2024-04-24T18:27:23.307 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53484.log.gz 2024-04-24T18:27:23.308 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60902.log 2024-04-24T18:27:23.308 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56872.log 2024-04-24T18:27:23.309 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65181.log 2024-04-24T18:27:23.310 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56872.log: /var/log/ceph/cluster1-client.admin.65181.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65181.log.gz 2024-04-24T18:27:23.310 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69798.log 2024-04-24T18:27:23.310 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56980.log.gz 2024-04-24T18:27:23.312 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58509.log.gz 2024-04-24T18:27:23.312 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58921.log.gz 2024-04-24T18:27:23.312 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59990.log.gz 2024-04-24T18:27:23.313 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60158.log.gz 2024-04-24T18:27:23.314 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69798.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69798.log.gz 2024-04-24T18:27:23.314 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60179.log.gz 2024-04-24T18:27:23.314 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62419.log 2024-04-24T18:27:23.314 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60137.log.gz 2024-04-24T18:27:23.314 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58828.log.gz 2024-04-24T18:27:23.315 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.62419.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59462.log.gz 2024-04-24T18:27:23.319 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.57344.log.gz 2024-04-24T18:27:23.319 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48811.log.gz 2024-04-24T18:27:23.320 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48613.log.gz 2024-04-24T18:27:23.323 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70101.log 2024-04-24T18:27:23.329 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60703.log.gz 2024-04-24T18:27:23.330 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49844.log.gz 2024-04-24T18:27:23.331 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49072.log.gz 2024-04-24T18:27:23.333 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70101.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49465.log 2024-04-24T18:27:23.333 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70101.log.gz 2024-04-24T18:27:23.334 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49434.log.gz 2024-04-24T18:27:23.336 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58412.log 2024-04-24T18:27:23.336 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47757.log.gz 2024-04-24T18:27:23.338 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50828.log.gz 2024-04-24T18:27:23.338 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50899.log.gz 2024-04-24T18:27:23.343 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58412.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54660.log.gz 2024-04-24T18:27:23.343 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56328.log 2024-04-24T18:27:23.345 INFO:teuthology.orchestra.run.smithi136.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56226.log.gz 2024-04-24T18:27:23.346 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56807.log.gz 2024-04-24T18:27:23.346 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56828.log.gz 2024-04-24T18:27:23.346 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46162.log.gz 2024-04-24T18:27:23.347 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56851.log.gz 2024-04-24T18:27:23.349 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.48293.log.gz 2024-04-24T18:27:23.359 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48655.log.gz 2024-04-24T18:27:23.359 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51966.log.gz 2024-04-24T18:27:23.359 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49114.log.gz 2024-04-24T18:27:23.359 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50658.log.gz 2024-04-24T18:27:23.359 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51333.log.gz 2024-04-24T18:27:23.360 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60493.log.gz 2024-04-24T18:27:23.360 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49908.log.gz 2024-04-24T18:27:23.370 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48698.log.gz 2024-04-24T18:27:23.370 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.56328.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49752.log.gz 2024-04-24T18:27:23.371 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61798.log 2024-04-24T18:27:23.371 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.55253.log.gz 2024-04-24T18:27:23.371 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59793.log 2024-04-24T18:27:23.371 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52030.log.gz 2024-04-24T18:27:23.371 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61122.log.gz 2024-04-24T18:27:23.371 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61101.log.gz 2024-04-24T18:27:23.371 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52662.log.gz 2024-04-24T18:27:23.371 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61798.log.gz 2024-04-24T18:27:23.371 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54130.log.gz 2024-04-24T18:27:23.371 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55465.log.gz 2024-04-24T18:27:23.372 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58043.log.gz 2024-04-24T18:27:23.372 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56037.log.gz 2024-04-24T18:27:23.372 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56693.log.gz 2024-04-24T18:27:23.372 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58114.log.gz 2024-04-24T18:27:23.372 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51987.log.gz 2024-04-24T18:27:23.372 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61187.log.gz 2024-04-24T18:27:23.372 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60944.log 2024-04-24T18:27:23.372 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59793.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58807.log.gz 2024-04-24T18:27:23.375 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69001.log 2024-04-24T18:27:23.375 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64128.log 2024-04-24T18:27:23.375 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69001.log.gz 2024-04-24T18:27:23.375 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60775.log.gz 2024-04-24T18:27:23.375 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60514.log.gz 2024-04-24T18:27:23.376 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61078.log.gz 2024-04-24T18:27:23.376 INFO:teuthology.orchestra.run.smithi136.stderr: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45643.log.gz 2024-04-24T18:27:23.377 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46686.log.gz 2024-04-24T18:27:23.377 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.48320.log.gz 2024-04-24T18:27:23.378 INFO:teuthology.orchestra.run.smithi136.stderr: 18.7% -- replaced with /var/log/ceph/cluster1-client.admin.54367.log.gz 2024-04-24T18:27:23.383 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47841.log 2024-04-24T18:27:23.384 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.64128.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64128.log.gz 2024-04-24T18:27:23.385 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54618.log 2024-04-24T18:27:23.386 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57569.log 2024-04-24T18:27:23.401 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46862.log 2024-04-24T18:27:23.402 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.57371.log.gz 2024-04-24T18:27:23.403 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69840.log 2024-04-24T18:27:23.404 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69840.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69840.log.gz 2024-04-24T18:27:23.405 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60986.log 2024-04-24T18:27:23.405 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60074.log 2024-04-24T18:27:23.406 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55090.log 2024-04-24T18:27:23.407 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.60074.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53694.log.gz 2024-04-24T18:27:23.408 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51916.log.gz 2024-04-24T18:27:23.409 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.55090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49022.log 2024-04-24T18:27:23.410 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.49022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63255.log 2024-04-24T18:27:23.411 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.63255.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.63255.log.gz 2024-04-24T18:27:23.412 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63309.log 2024-04-24T18:27:23.413 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.63309.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.63309.log.gz 2024-04-24T18:27:23.413 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53799.log 2024-04-24T18:27:23.416 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59616.log 2024-04-24T18:27:23.425 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48470.log 2024-04-24T18:27:23.429 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61995.log 2024-04-24T18:27:23.430 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54017.log.gz 2024-04-24T18:27:23.440 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48470.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54745.log.gz 2024-04-24T18:27:23.440 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55111.log.gz 2024-04-24T18:27:23.442 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56672.log.gz 2024-04-24T18:27:23.452 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60074.log.gz 2024-04-24T18:27:23.452 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54703.log.gz 2024-04-24T18:27:23.453 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58764.log.gz 2024-04-24T18:27:23.453 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59616.log.gz 2024-04-24T18:27:23.453 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61995.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61995.log.gz 2024-04-24T18:27:23.453 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44188.log 2024-04-24T18:27:23.453 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44188.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44188.log.gz 2024-04-24T18:27:23.453 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69891.log 2024-04-24T18:27:23.453 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69891.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69891.log.gz 2024-04-24T18:27:23.453 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60374.log 2024-04-24T18:27:23.456 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59793.log.gz 2024-04-24T18:27:23.458 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60944.log.gz 2024-04-24T18:27:23.459 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59919.log.gz 2024-04-24T18:27:23.466 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59814.log.gz 2024-04-24T18:27:23.471 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46058.log 2024-04-24T18:27:23.476 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.58305.log.gz 2024-04-24T18:27:23.476 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster2-client.admin.51357.log.gz 2024-04-24T18:27:23.477 INFO:teuthology.orchestra.run.smithi136.stderr: 53.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45677.log.gz 2024-04-24T18:27:23.486 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.56328.log.gz 2024-04-24T18:27:23.486 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48470.log.gz 2024-04-24T18:27:23.486 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47841.log.gz 2024-04-24T18:27:23.486 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46862.log.gz 2024-04-24T18:27:23.486 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50015.log.gz 2024-04-24T18:27:23.486 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.57398.log.gz 2024-04-24T18:27:23.486 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.47084.log.gz 2024-04-24T18:27:23.486 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46883.log.gz 2024-04-24T18:27:23.486 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47778.log.gz 2024-04-24T18:27:23.487 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68457.log 2024-04-24T18:27:23.487 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50722.log.gz 2024-04-24T18:27:23.487 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.68457.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.68457.log.gz 2024-04-24T18:27:23.487 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53077.log 2024-04-24T18:27:23.488 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.53077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55974.log 2024-04-24T18:27:23.489 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55090.log.gz 2024-04-24T18:27:23.490 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54597.log.gz 2024-04-24T18:27:23.492 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52008.log.gz 2024-04-24T18:27:23.493 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55692.log.gz 2024-04-24T18:27:23.502 INFO:teuthology.orchestra.run.smithi136.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57270.log.gz 2024-04-24T18:27:23.502 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58177.log.gz 2024-04-24T18:27:23.502 INFO:teuthology.orchestra.run.smithi136.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55498.log.gz 2024-04-24T18:27:23.502 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60796.log 2024-04-24T18:27:23.503 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56872.log.gz 2024-04-24T18:27:23.503 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60116.log.gz 2024-04-24T18:27:23.503 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55903.log.gz 2024-04-24T18:27:23.503 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57739.log.gz 2024-04-24T18:27:23.503 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59162.log.gz 2024-04-24T18:27:23.503 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.61337.log.gz 2024-04-24T18:27:23.503 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60796.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59405.log 2024-04-24T18:27:23.504 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.59405.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.59405.log.gz 2024-04-24T18:27:23.504 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67853.log 2024-04-24T18:27:23.504 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67853.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56164.log.gz 2024-04-24T18:27:23.504 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67853.log.gz 2024-04-24T18:27:23.505 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65036.log 2024-04-24T18:27:23.506 INFO:teuthology.orchestra.run.smithi136.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60245.log.gz 2024-04-24T18:27:23.507 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47138.log 2024-04-24T18:27:23.508 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.65036.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65036.log.gz 2024-04-24T18:27:23.508 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.47138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51130.log 2024-04-24T18:27:23.510 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45545.log 2024-04-24T18:27:23.511 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.51130.log: /var/log/ceph/cluster1-client.admin.45545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56142.log 2024-04-24T18:27:23.512 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52449.log 2024-04-24T18:27:23.513 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.56142.log: /var/log/ceph/cluster1-client.mirror.52449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57425.log 2024-04-24T18:27:23.514 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.57425.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47820.log 2024-04-24T18:27:23.515 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61390.log 2024-04-24T18:27:23.520 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64797.log 2024-04-24T18:27:23.521 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70533.log 2024-04-24T18:27:23.521 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.55231.log.gz 2024-04-24T18:27:23.524 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64797.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50107.log.gz 2024-04-24T18:27:23.532 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster1-client.admin.50387.log.gz 2024-04-24T18:27:23.533 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.47138.log.gz 2024-04-24T18:27:23.533 INFO:teuthology.orchestra.run.smithi136.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.47062.log.gz 2024-04-24T18:27:23.533 INFO:teuthology.orchestra.run.smithi136.stderr: 58.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45867.log.gz 2024-04-24T18:27:23.533 INFO:teuthology.orchestra.run.smithi136.stderr: 56.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45884.log.gz 2024-04-24T18:27:23.533 INFO:teuthology.orchestra.run.smithi136.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47027.log.gz 2024-04-24T18:27:23.533 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46364.log.gz 2024-04-24T18:27:23.533 INFO:teuthology.orchestra.run.smithi136.stderr: 5.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45609.log.gz 2024-04-24T18:27:23.533 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46058.log.gz 2024-04-24T18:27:23.534 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64797.log.gz 2024-04-24T18:27:23.535 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.70533.log: 18.1% -- replaced with /var/log/ceph/cluster1-client.admin.70533.log.gz 2024-04-24T18:27:23.535 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48017.log 2024-04-24T18:27:23.536 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.48017.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52195.log 2024-04-24T18:27:23.537 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.52195.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52195.log.gz 2024-04-24T18:27:23.537 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40462.log 2024-04-24T18:27:23.538 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.40462.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40462.log.gz 2024-04-24T18:27:23.538 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57761.log 2024-04-24T18:27:23.539 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60753.log 2024-04-24T18:27:23.540 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48397.log 2024-04-24T18:27:23.540 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.48397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.45957.log 2024-04-24T18:27:23.541 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47609.log 2024-04-24T18:27:23.543 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49465.log.gz 2024-04-24T18:27:23.543 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50036.log.gz 2024-04-24T18:27:23.545 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.0.45957.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48853.log.gz 2024-04-24T18:27:23.546 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45660.log 2024-04-24T18:27:23.547 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.45660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55642.log 2024-04-24T18:27:23.548 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46665.log 2024-04-24T18:27:23.549 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58359.log 2024-04-24T18:27:23.550 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52943.log 2024-04-24T18:27:23.551 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.58359.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51130.log.gz 2024-04-24T18:27:23.555 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52943.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50057.log 2024-04-24T18:27:23.556 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.50057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69819.log 2024-04-24T18:27:23.558 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69819.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69819.log.gz 2024-04-24T18:27:23.558 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69400.log 2024-04-24T18:27:23.559 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.69400.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.69400.log.gz 2024-04-24T18:27:23.559 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71695.log 2024-04-24T18:27:23.560 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.71695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66531.log 2024-04-24T18:27:23.560 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.71695.log.gz 2024-04-24T18:27:23.562 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66531.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66531.log.gz 2024-04-24T18:27:23.562 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52733.log 2024-04-24T18:27:23.562 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52733.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56376.log 2024-04-24T18:27:23.564 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48916.log 2024-04-24T18:27:23.564 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.56376.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.56376.log.gz 2024-04-24T18:27:23.564 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57853.log 2024-04-24T18:27:23.566 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68917.log 2024-04-24T18:27:23.567 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68917.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68917.log.gz 2024-04-24T18:27:23.567 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57451.log 2024-04-24T18:27:23.568 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57451.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57938.log 2024-04-24T18:27:23.569 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51612.log.gz 2024-04-24T18:27:23.569 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52449.log.gz 2024-04-24T18:27:23.570 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66381.log 2024-04-24T18:27:23.571 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54618.log.gz 2024-04-24T18:27:23.572 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49866.log.gz 2024-04-24T18:27:23.575 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58611.log.gz 2024-04-24T18:27:23.576 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56610.log.gz 2024-04-24T18:27:23.578 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56142.log.gz 2024-04-24T18:27:23.579 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57875.log.gz 2024-04-24T18:27:23.590 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57938.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58786.log.gz 2024-04-24T18:27:23.593 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59708.log.gz 2024-04-24T18:27:23.595 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59835.log.gz 2024-04-24T18:27:23.596 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53077.log.gz 2024-04-24T18:27:23.598 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66381.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.61364.log.gz 2024-04-24T18:27:23.598 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66381.log.gz 2024-04-24T18:27:23.598 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60011.log 2024-04-24T18:27:23.600 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50941.log.gz 2024-04-24T18:27:23.601 INFO:teuthology.orchestra.run.smithi136.stderr: 18.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.45957.log.gz 2024-04-24T18:27:23.609 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61015.log.gz 2024-04-24T18:27:23.609 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60986.log.gz 2024-04-24T18:27:23.609 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.53321.log.gz 2024-04-24T18:27:23.609 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.60011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69454.log 2024-04-24T18:27:23.621 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.69454.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.69454.log.gz 2024-04-24T18:27:23.621 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47630.log 2024-04-24T18:27:23.629 INFO:teuthology.orchestra.run.smithi136.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.58359.log.gz 2024-04-24T18:27:23.629 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55642.log.gz 2024-04-24T18:27:23.629 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48916.log.gz 2024-04-24T18:27:23.629 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53014.log.gz 2024-04-24T18:27:23.629 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52733.log.gz 2024-04-24T18:27:23.629 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49022.log.gz 2024-04-24T18:27:23.629 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50057.log.gz 2024-04-24T18:27:23.630 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66820.log 2024-04-24T18:27:23.633 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66820.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53912.log.gz 2024-04-24T18:27:23.633 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66820.log.gz 2024-04-24T18:27:23.633 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66062.log 2024-04-24T18:27:23.634 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.66062.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66062.log.gz 2024-04-24T18:27:23.634 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57146.log 2024-04-24T18:27:23.634 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71931.log 2024-04-24T18:27:23.635 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.71931.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71931.log.gz 2024-04-24T18:27:23.636 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48958.log 2024-04-24T18:27:23.636 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48958.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47558.log 2024-04-24T18:27:23.637 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65057.log 2024-04-24T18:27:23.639 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.65057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63718.log 2024-04-24T18:27:23.639 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65057.log.gz 2024-04-24T18:27:23.641 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63718.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63718.log.gz 2024-04-24T18:27:23.641 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71474.log 2024-04-24T18:27:23.642 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69200.log 2024-04-24T18:27:23.642 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.71474.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.71474.log.gz 2024-04-24T18:27:23.644 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69200.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69200.log.gz 2024-04-24T18:27:23.644 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40731.log 2024-04-24T18:27:23.645 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.40731.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40731.log.gz 2024-04-24T18:27:23.645 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61848.log 2024-04-24T18:27:23.647 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61848.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61848.log.gz 2024-04-24T18:27:23.647 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67811.log 2024-04-24T18:27:23.648 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67811.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67811.log.gz 2024-04-24T18:27:23.649 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54059.log 2024-04-24T18:27:23.649 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.54059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53736.log 2024-04-24T18:27:23.650 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66862.log 2024-04-24T18:27:23.652 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66862.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66862.log.gz 2024-04-24T18:27:23.652 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62109.log 2024-04-24T18:27:23.654 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.62109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51633.log 2024-04-24T18:27:23.654 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62109.log.gz 2024-04-24T18:27:23.669 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56917.log.gz 2024-04-24T18:27:23.669 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57569.log.gz 2024-04-24T18:27:23.669 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57896.log.gz 2024-04-24T18:27:23.669 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55819.log.gz 2024-04-24T18:27:23.669 INFO:teuthology.orchestra.run.smithi136.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58262.log.gz 2024-04-24T18:27:23.669 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66012.log 2024-04-24T18:27:23.669 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.56301.log.gz 2024-04-24T18:27:23.679 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58085.log.gz 2024-04-24T18:27:23.679 INFO:teuthology.orchestra.run.smithi136.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59243.log.gz 2024-04-24T18:27:23.679 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.66012.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66012.log.gz 2024-04-24T18:27:23.679 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59877.log 2024-04-24T18:27:23.681 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52327.log.gz 2024-04-24T18:27:23.692 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60796.log.gz 2024-04-24T18:27:23.692 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61036.log.gz 2024-04-24T18:27:23.693 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44487.log 2024-04-24T18:27:23.700 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47870.log.gz 2024-04-24T18:27:23.705 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44487.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44487.log.gz 2024-04-24T18:27:23.705 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56247.log 2024-04-24T18:27:23.706 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.56247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60465.log 2024-04-24T18:27:23.709 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56714.log 2024-04-24T18:27:23.710 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60267.log 2024-04-24T18:27:23.711 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.60267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52817.log 2024-04-24T18:27:23.712 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58722.log 2024-04-24T18:27:23.712 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51874.log 2024-04-24T18:27:23.714 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48017.log.gz 2024-04-24T18:27:23.714 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48958.log.gz 2024-04-24T18:27:23.715 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70080.log 2024-04-24T18:27:23.717 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70080.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70080.log.gz 2024-04-24T18:27:23.717 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58001.log 2024-04-24T18:27:23.718 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.58001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61143.log 2024-04-24T18:27:23.719 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.61143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70632.log 2024-04-24T18:27:23.720 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70632.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51945.log 2024-04-24T18:27:23.721 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70632.log.gz 2024-04-24T18:27:23.722 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52775.log 2024-04-24T18:27:23.722 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59898.log 2024-04-24T18:27:23.723 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47286.log 2024-04-24T18:27:23.724 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50474.log 2024-04-24T18:27:23.725 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60348.log 2024-04-24T18:27:23.726 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48676.log 2024-04-24T18:27:23.727 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.60348.log: /var/log/ceph/cluster1-client.mirror.48676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50149.log 2024-04-24T18:27:23.728 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.50149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44027.log 2024-04-24T18:27:23.729 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44027.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44027.log.gz 2024-04-24T18:27:23.729 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68505.log 2024-04-24T18:27:23.730 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.68505.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.68505.log.gz 2024-04-24T18:27:23.730 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64902.log 2024-04-24T18:27:23.732 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64902.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64902.log.gz 2024-04-24T18:27:23.732 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67748.log 2024-04-24T18:27:23.734 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67748.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67748.log.gz 2024-04-24T18:27:23.734 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57824.log 2024-04-24T18:27:23.735 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49929.log 2024-04-24T18:27:23.735 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.57425.log.gz 2024-04-24T18:27:23.736 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57824.log: /var/log/ceph/cluster2-client.mirror.49929.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51633.log.gz 2024-04-24T18:27:23.736 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61412.log 2024-04-24T18:27:23.737 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.61412.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61412.log.gz 2024-04-24T18:27:23.737 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.50256.log 2024-04-24T18:27:23.738 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.50256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71172.log 2024-04-24T18:27:23.740 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71172.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71172.log.gz 2024-04-24T18:27:23.740 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72353.log 2024-04-24T18:27:23.741 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72353.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72353.log.gz 2024-04-24T18:27:23.741 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48895.log 2024-04-24T18:27:23.741 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49815.log 2024-04-24T18:27:23.742 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49815.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59729.log 2024-04-24T18:27:23.743 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46602.log 2024-04-24T18:27:23.744 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70604.log 2024-04-24T18:27:23.746 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70604.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70604.log.gz 2024-04-24T18:27:23.746 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64107.log 2024-04-24T18:27:23.748 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.64107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63655.log 2024-04-24T18:27:23.748 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64107.log.gz 2024-04-24T18:27:23.750 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63655.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63655.log.gz 2024-04-24T18:27:23.750 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46644.log 2024-04-24T18:27:23.751 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72069.log 2024-04-24T18:27:23.752 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72069.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72069.log.gz 2024-04-24T18:27:23.752 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60682.log 2024-04-24T18:27:23.753 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51874.log.gz 2024-04-24T18:27:23.754 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46925.log.gz 2024-04-24T18:27:23.755 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69373.log 2024-04-24T18:27:23.756 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47452.log.gz 2024-04-24T18:27:23.757 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51945.log.gz 2024-04-24T18:27:23.757 INFO:teuthology.orchestra.run.smithi136.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.60348.log.gz 2024-04-24T18:27:23.758 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60682.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52817.log.gz 2024-04-24T18:27:23.759 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.69373.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.69373.log.gz 2024-04-24T18:27:23.760 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58198.log 2024-04-24T18:27:23.763 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46778.log 2024-04-24T18:27:23.767 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53799.log.gz 2024-04-24T18:27:23.769 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54059.log.gz 2024-04-24T18:27:23.772 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66913.log 2024-04-24T18:27:23.781 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.58198.log: /var/log/ceph/cluster1-client.mirror.66913.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66913.log.gz 2024-04-24T18:27:23.781 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53294.log 2024-04-24T18:27:23.782 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.53294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63762.log 2024-04-24T18:27:23.784 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63762.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57530.log 2024-04-24T18:27:23.784 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63762.log.gz 2024-04-24T18:27:23.785 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.70402.log 2024-04-24T18:27:23.786 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.70402.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.70402.log.gz 2024-04-24T18:27:23.786 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48244.log 2024-04-24T18:27:23.787 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.48244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58659.log 2024-04-24T18:27:23.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53035.log 2024-04-24T18:27:23.789 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.53035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58022.log 2024-04-24T18:27:23.790 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.58022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60838.log 2024-04-24T18:27:23.792 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49973.log 2024-04-24T18:27:23.792 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55974.log.gz 2024-04-24T18:27:23.794 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60838.log: /var/log/ceph/cluster2-client.mirror.49973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59750.log 2024-04-24T18:27:23.794 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51769.log 2024-04-24T18:27:23.795 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45041.log 2024-04-24T18:27:23.796 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.45041.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.45041.log.gz 2024-04-24T18:27:23.796 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56498.log 2024-04-24T18:27:23.797 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57093.log 2024-04-24T18:27:23.798 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61165.log 2024-04-24T18:27:23.799 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.61165.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53589.log 2024-04-24T18:27:23.800 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45179.log 2024-04-24T18:27:23.801 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47579.log 2024-04-24T18:27:23.803 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.45179.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57761.log.gz 2024-04-24T18:27:23.804 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59898.log.gz 2024-04-24T18:27:23.805 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57824.log.gz 2024-04-24T18:27:23.805 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59877.log.gz 2024-04-24T18:27:23.806 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58539.log.gz 2024-04-24T18:27:23.813 INFO:teuthology.orchestra.run.smithi136.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.50256.log.gz 2024-04-24T18:27:23.814 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58561.log.gz 2024-04-24T18:27:23.816 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46602.log.gz 2024-04-24T18:27:23.821 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46778.log.gz 2024-04-24T18:27:23.821 INFO:teuthology.orchestra.run.smithi136.stderr: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45660.log.gz 2024-04-24T18:27:23.821 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60095.log.gz 2024-04-24T18:27:23.821 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49626.log 2024-04-24T18:27:23.821 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49973.log.gz 2024-04-24T18:27:23.822 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48895.log.gz 2024-04-24T18:27:23.822 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.45179.log.gz 2024-04-24T18:27:23.830 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57718.log 2024-04-24T18:27:23.851 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57718.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66167.log 2024-04-24T18:27:23.854 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.66167.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66167.log.gz 2024-04-24T18:27:23.854 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70982.log 2024-04-24T18:27:23.855 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57022.log.gz 2024-04-24T18:27:23.857 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57146.log.gz 2024-04-24T18:27:23.859 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70982.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57853.log.gz 2024-04-24T18:27:23.860 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58001.log.gz 2024-04-24T18:27:23.861 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58198.log.gz 2024-04-24T18:27:23.864 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66998.log 2024-04-24T18:27:23.864 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58412.log.gz 2024-04-24T18:27:23.864 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49929.log.gz 2024-04-24T18:27:23.866 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50149.log.gz 2024-04-24T18:27:23.869 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.55334.log.gz 2024-04-24T18:27:23.869 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60753.log.gz 2024-04-24T18:27:23.870 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60838.log.gz 2024-04-24T18:27:23.870 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70982.log.gz 2024-04-24T18:27:23.880 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46665.log.gz 2024-04-24T18:27:23.880 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61390.log.gz 2024-04-24T18:27:23.882 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66998.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66998.log.gz 2024-04-24T18:27:23.882 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48395.log 2024-04-24T18:27:23.893 INFO:teuthology.orchestra.run.smithi042.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-04-24T18:27:23.893 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.48395.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.48395.log.gz 2024-04-24T18:27:23.894 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50637.log 2024-04-24T18:27:23.901 INFO:teuthology.orchestra.run.smithi042.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-04-24T18:27:23.902 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.54236.log.gz 2024-04-24T18:27:23.909 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69158.log 2024-04-24T18:27:23.916 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69158.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53035.log.gz 2024-04-24T18:27:23.916 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69158.log.gz 2024-04-24T18:27:23.916 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65667.log 2024-04-24T18:27:23.918 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65667.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.49275.log.gz 2024-04-24T18:27:23.918 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65667.log.gz 2024-04-24T18:27:23.918 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58283.log 2024-04-24T18:27:23.919 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.58283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56222.log 2024-04-24T18:27:23.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.56222.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.56222.log.gz 2024-04-24T18:27:23.920 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63441.log 2024-04-24T18:27:23.922 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63441.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63441.log.gz 2024-04-24T18:27:23.922 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49731.log 2024-04-24T18:27:23.922 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61249.log 2024-04-24T18:27:23.923 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.61249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70278.log 2024-04-24T18:27:23.925 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70278.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70278.log.gz 2024-04-24T18:27:23.926 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53267.log 2024-04-24T18:27:23.926 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49626.log.gz 2024-04-24T18:27:23.927 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.53267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48347.log 2024-04-24T18:27:23.928 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49668.log.gz 2024-04-24T18:27:23.928 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.48347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50363.log 2024-04-24T18:27:23.938 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49815.log.gz 2024-04-24T18:27:23.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72445.log 2024-04-24T18:27:23.939 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47558.log.gz 2024-04-24T18:27:23.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72445.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72445.log.gz 2024-04-24T18:27:23.939 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55882.log 2024-04-24T18:27:23.940 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55755.log 2024-04-24T18:27:23.952 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64527.log 2024-04-24T18:27:23.954 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62983.log 2024-04-24T18:27:23.954 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64527.log.gz 2024-04-24T18:27:23.956 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67483.log 2024-04-24T18:27:23.958 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58022.log.gz 2024-04-24T18:27:23.958 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57959.log.gz 2024-04-24T18:27:23.959 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60011.log.gz 2024-04-24T18:27:23.962 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60682.log.gz 2024-04-24T18:27:23.962 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62983.log.gz 2024-04-24T18:27:23.962 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60902.log.gz 2024-04-24T18:27:23.979 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.67483.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.56247.log.gz 2024-04-24T18:27:23.982 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53736.log.gz 2024-04-24T18:27:23.984 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63741.log 2024-04-24T18:27:23.984 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50474.log.gz 2024-04-24T18:27:23.995 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63741.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63741.log.gz 2024-04-24T18:27:23.995 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61651.log 2024-04-24T18:27:23.995 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56714.log.gz 2024-04-24T18:27:24.017 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61651.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61651.log.gz 2024-04-24T18:27:24.017 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64350.log 2024-04-24T18:27:24.028 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64350.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65801.log 2024-04-24T18:27:24.028 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64350.log.gz 2024-04-24T18:27:24.039 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.58283.log.gz 2024-04-24T18:27:24.050 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64374.log 2024-04-24T18:27:24.050 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65801.log.gz 2024-04-24T18:27:24.051 INFO:teuthology.orchestra.run.smithi136.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.48244.log.gz 2024-04-24T18:27:24.060 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.53267.log.gz 2024-04-24T18:27:24.060 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47820.log.gz 2024-04-24T18:27:24.060 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50637.log.gz 2024-04-24T18:27:24.060 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47286.log.gz 2024-04-24T18:27:24.071 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49731.log.gz 2024-04-24T18:27:24.071 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.64374.log: 18.7% -- replaced with /var/log/ceph/cluster1-client.admin.64374.log.gz 2024-04-24T18:27:24.071 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54458.log 2024-04-24T18:27:24.082 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55882.log.gz 2024-04-24T18:27:24.082 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57697.log.gz 2024-04-24T18:27:24.082 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57530.log.gz 2024-04-24T18:27:24.082 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57938.log.gz 2024-04-24T18:27:24.082 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55755.log.gz 2024-04-24T18:27:24.083 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57093.log.gz 2024-04-24T18:27:24.083 INFO:teuthology.orchestra.run.smithi136.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.45545.log.gz 2024-04-24T18:27:24.093 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52775.log.gz 2024-04-24T18:27:24.093 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60724.log.gz 2024-04-24T18:27:24.093 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54458.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57718.log.gz 2024-04-24T18:27:24.093 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57001.log 2024-04-24T18:27:24.104 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61165.log.gz 2024-04-24T18:27:24.104 INFO:teuthology.orchestra.run.smithi136.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50363.log.gz 2024-04-24T18:27:24.104 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54317.log 2024-04-24T18:27:24.112 INFO:teuthology.orchestra.run.smithi136.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.53294.log.gz 2024-04-24T18:27:24.122 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47630.log.gz 2024-04-24T18:27:24.133 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.60267.log.gz 2024-04-24T18:27:24.133 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.54317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51067.log 2024-04-24T18:27:24.133 INFO:teuthology.orchestra.run.smithi136.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.48347.log.gz 2024-04-24T18:27:24.144 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.51067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65907.log 2024-04-24T18:27:24.155 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54458.log.gz 2024-04-24T18:27:24.155 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58659.log.gz 2024-04-24T18:27:24.166 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65907.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65907.log.gz 2024-04-24T18:27:24.166 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48832.log 2024-04-24T18:27:24.168 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59112.log.gz 2024-04-24T18:27:24.169 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48832.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60465.log.gz 2024-04-24T18:27:24.169 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52859.log 2024-04-24T18:27:24.171 INFO:teuthology.orchestra.run.smithi136.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61249.log.gz 2024-04-24T18:27:24.178 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57001.log.gz 2024-04-24T18:27:24.179 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60374.log.gz 2024-04-24T18:27:24.182 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-04-24T18:27:24.202 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63336.log 2024-04-24T18:27:24.225 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.63336.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.63336.log.gz 2024-04-24T18:27:24.226 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47715.log 2024-04-24T18:27:24.247 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55734.log 2024-04-24T18:27:24.256 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47579.log.gz 2024-04-24T18:27:24.257 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47609.log.gz 2024-04-24T18:27:24.259 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46644.log.gz 2024-04-24T18:27:24.280 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.54317.log.gz 2024-04-24T18:27:24.280 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47715.log.gz 2024-04-24T18:27:24.280 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48676.log.gz 2024-04-24T18:27:24.281 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53673.log.gz 2024-04-24T18:27:24.289 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60965.log 2024-04-24T18:27:24.302 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56938.log 2024-04-24T18:27:24.310 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50701.log 2024-04-24T18:27:24.314 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48832.log.gz 2024-04-24T18:27:24.326 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47191.log 2024-04-24T18:27:24.328 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53589.log.gz 2024-04-24T18:27:24.330 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56938.log.gz 2024-04-24T18:27:24.339 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47191.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56274.log 2024-04-24T18:27:24.353 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.56274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58386.log 2024-04-24T18:27:24.377 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.58386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44395.log 2024-04-24T18:27:24.391 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44395.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44395.log.gz 2024-04-24T18:27:24.391 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44533.log 2024-04-24T18:27:24.400 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.44533.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44533.log.gz 2024-04-24T18:27:24.401 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48553.log 2024-04-24T18:27:24.425 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48553.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55069.log 2024-04-24T18:27:24.426 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47191.log.gz 2024-04-24T18:27:24.427 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56498.log.gz 2024-04-24T18:27:24.429 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57451.log.gz 2024-04-24T18:27:24.430 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58722.log.gz 2024-04-24T18:27:24.439 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52859.log.gz 2024-04-24T18:27:24.439 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51067.log.gz 2024-04-24T18:27:24.439 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59729.log.gz 2024-04-24T18:27:24.440 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51769.log.gz 2024-04-24T18:27:24.449 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.55069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44165.log 2024-04-24T18:27:24.460 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44165.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44165.log.gz 2024-04-24T18:27:24.460 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62920.log 2024-04-24T18:27:24.476 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62257.log 2024-04-24T18:27:24.476 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62920.log.gz 2024-04-24T18:27:24.488 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.62257.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62257.log.gz 2024-04-24T18:27:24.488 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.70429.log 2024-04-24T18:27:24.509 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.70429.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.70429.log.gz 2024-04-24T18:27:24.509 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49887.log 2024-04-24T18:27:24.526 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52220.log 2024-04-24T18:27:24.540 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.52220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51895.log 2024-04-24T18:27:24.548 INFO:teuthology.orchestra.run.smithi136.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.58386.log.gz 2024-04-24T18:27:24.558 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50701.log.gz 2024-04-24T18:27:24.569 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61057.log 2024-04-24T18:27:24.580 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.61057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47799.log 2024-04-24T18:27:24.580 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster1-client.admin.48397.log.gz 2024-04-24T18:27:24.591 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47388.log.gz 2024-04-24T18:27:24.591 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70890.log 2024-04-24T18:27:24.603 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52943.log.gz 2024-04-24T18:27:24.603 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51895.log.gz 2024-04-24T18:27:24.613 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49887.log.gz 2024-04-24T18:27:24.613 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55069.log.gz 2024-04-24T18:27:24.613 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.56274.log.gz 2024-04-24T18:27:24.624 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48553.log.gz 2024-04-24T18:27:24.639 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70890.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70890.log.gz 2024-04-24T18:27:24.639 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55924.log 2024-04-24T18:27:24.639 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49388.log 2024-04-24T18:27:24.651 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.49388.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52135.log 2024-04-24T18:27:24.664 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61143.log.gz 2024-04-24T18:27:24.664 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.52135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65864.log 2024-04-24T18:27:24.675 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65864.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65864.log.gz 2024-04-24T18:27:24.675 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59658.log 2024-04-24T18:27:24.692 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50128.log 2024-04-24T18:27:24.715 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.50128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54921.log 2024-04-24T18:27:24.729 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63697.log 2024-04-24T18:27:24.741 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63697.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63697.log.gz 2024-04-24T18:27:24.741 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54431.log 2024-04-24T18:27:24.755 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49647.log 2024-04-24T18:27:24.767 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55861.log 2024-04-24T18:27:24.779 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55861.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64692.log 2024-04-24T18:27:24.789 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59750.log.gz 2024-04-24T18:27:24.802 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64692.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64692.log.gz 2024-04-24T18:27:24.802 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63897.log 2024-04-24T18:27:24.813 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65015.log 2024-04-24T18:27:24.824 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63897.log.gz 2024-04-24T18:27:24.849 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.65015.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65015.log.gz 2024-04-24T18:27:24.849 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65320.log 2024-04-24T18:27:24.855 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.65320.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.65320.log.gz 2024-04-24T18:27:24.855 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56786.log 2024-04-24T18:27:24.874 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61630.log 2024-04-24T18:27:24.900 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61630.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61630.log.gz 2024-04-24T18:27:24.900 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62495.log 2024-04-24T18:27:24.914 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62495.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62495.log.gz 2024-04-24T18:27:24.914 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-04-24T18:27:24.925 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49302.log 2024-04-24T18:27:24.939 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.49302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51464.log 2024-04-24T18:27:24.962 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67370.log 2024-04-24T18:27:24.976 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.67370.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.67370.log.gz 2024-04-24T18:27:24.977 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56448.log 2024-04-24T18:27:24.987 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49647.log.gz 2024-04-24T18:27:24.987 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52135.log.gz 2024-04-24T18:27:24.998 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70320.log 2024-04-24T18:27:24.998 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56786.log.gz 2024-04-24T18:27:24.998 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54921.log.gz 2024-04-24T18:27:24.998 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61057.log.gz 2024-04-24T18:27:25.009 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50128.log.gz 2024-04-24T18:27:25.015 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62408.log 2024-04-24T18:27:25.015 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70320.log.gz 2024-04-24T18:27:25.026 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.62408.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62408.log.gz 2024-04-24T18:27:25.026 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59856.log 2024-04-24T18:27:25.037 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55734.log.gz 2024-04-24T18:27:25.043 INFO:teuthology.orchestra.run.smithi136.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.52220.log.gz 2024-04-24T18:27:25.050 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59856.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54431.log.gz 2024-04-24T18:27:25.050 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54088.log 2024-04-24T18:27:25.050 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60965.log.gz 2024-04-24T18:27:25.061 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.54088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46518.log 2024-04-24T18:27:25.061 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47799.log.gz 2024-04-24T18:27:25.074 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55006.log 2024-04-24T18:27:25.085 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.55006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63825.log 2024-04-24T18:27:25.108 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63825.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63825.log.gz 2024-04-24T18:27:25.108 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54682.log 2024-04-24T18:27:25.116 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54088.log.gz 2024-04-24T18:27:25.123 INFO:teuthology.orchestra.run.smithi136.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.49302.log.gz 2024-04-24T18:27:25.123 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62658.log 2024-04-24T18:27:25.123 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51464.log.gz 2024-04-24T18:27:25.134 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59658.log.gz 2024-04-24T18:27:25.134 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55861.log.gz 2024-04-24T18:27:25.134 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55924.log.gz 2024-04-24T18:27:25.135 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55006.log.gz 2024-04-24T18:27:25.154 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56448.log.gz 2024-04-24T18:27:25.175 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62658.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62658.log.gz 2024-04-24T18:27:25.175 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65751.log 2024-04-24T18:27:25.193 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54682.log.gz 2024-04-24T18:27:25.200 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59856.log.gz 2024-04-24T18:27:25.200 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65751.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65751.log.gz 2024-04-24T18:27:25.201 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70847.log 2024-04-24T18:27:25.215 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70847.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70847.log.gz 2024-04-24T18:27:25.225 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65970.log 2024-04-24T18:27:25.242 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67916.log 2024-04-24T18:27:25.242 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65970.log.gz 2024-04-24T18:27:25.257 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.67916.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67916.log.gz 2024-04-24T18:27:25.267 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57475.log 2024-04-24T18:27:25.278 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46518.log.gz 2024-04-24T18:27:25.284 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.57475.log: 18.1% -- replaced with /var/log/ceph/cluster2-client.admin.57475.log.gz 2024-04-24T18:27:25.284 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70381.log 2024-04-24T18:27:25.284 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70381.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70381.log.gz 2024-04-24T18:27:25.284 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62856.log 2024-04-24T18:27:25.316 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62856.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62856.log.gz 2024-04-24T18:27:25.316 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43797.log 2024-04-24T18:27:25.327 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.43797.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43797.log.gz 2024-04-24T18:27:25.327 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57167.log 2024-04-24T18:27:25.345 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.57167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53870.log 2024-04-24T18:27:25.359 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53870.log: gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-04-24T18:27:25.380 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61477.log 2024-04-24T18:27:25.403 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61477.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61477.log.gz 2024-04-24T18:27:25.403 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48634.log 2024-04-24T18:27:25.414 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71603.log 2024-04-24T18:27:25.425 INFO:teuthology.orchestra.run.smithi136.stderr: 86.8% -- replaced with /var/log/ceph/cluster2.log.gz 2024-04-24T18:27:25.436 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.71603.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71603.log.gz 2024-04-24T18:27:25.436 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65099.log 2024-04-24T18:27:25.447 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.65099.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65099.log.gz 2024-04-24T18:27:25.447 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52578.log 2024-04-24T18:27:25.458 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57167.log.gz 2024-04-24T18:27:25.469 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.52578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48497.log 2024-04-24T18:27:25.482 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49379.log 2024-04-24T18:27:25.511 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.49379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51109.log 2024-04-24T18:27:25.511 INFO:teuthology.orchestra.run.smithi136.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53870.log.gz 2024-04-24T18:27:25.524 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.51109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71411.log 2024-04-24T18:27:25.538 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71411.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71411.log.gz 2024-04-24T18:27:25.538 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47165.log 2024-04-24T18:27:25.552 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.47165.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53426.log 2024-04-24T18:27:25.565 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59495.log 2024-04-24T18:27:25.575 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48634.log.gz 2024-04-24T18:27:25.575 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52578.log.gz 2024-04-24T18:27:25.590 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.59495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57594.log 2024-04-24T18:27:25.601 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69427.log 2024-04-24T18:27:25.620 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.69427.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.69427.log.gz 2024-04-24T18:27:25.621 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66301.log 2024-04-24T18:27:25.644 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.66301.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.66301.log.gz 2024-04-24T18:27:25.655 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45478.log 2024-04-24T18:27:25.656 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.45478.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45478.log.gz 2024-04-24T18:27:25.656 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67103.log 2024-04-24T18:27:25.680 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.67103.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67103.log.gz 2024-04-24T18:27:25.680 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51187.log 2024-04-24T18:27:25.691 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.51187.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51187.log.gz 2024-04-24T18:27:25.691 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53098.log 2024-04-24T18:27:25.702 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.53098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50078.log 2024-04-24T18:27:25.702 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51109.log.gz 2024-04-24T18:27:25.713 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.50078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71407.log 2024-04-24T18:27:25.713 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53426.log.gz 2024-04-24T18:27:25.724 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.71407.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.71407.log.gz 2024-04-24T18:27:25.724 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69072.log 2024-04-24T18:27:25.735 INFO:teuthology.orchestra.run.smithi136.stderr: 17.5% -- replaced with /var/log/ceph/cluster2-client.admin.49379.log.gz 2024-04-24T18:27:25.735 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.69072.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69072.log.gz 2024-04-24T18:27:25.735 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66757.log 2024-04-24T18:27:25.762 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.47165.log.gz 2024-04-24T18:27:25.768 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57594.log.gz 2024-04-24T18:27:25.778 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59495.log.gz 2024-04-24T18:27:25.778 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66757.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66757.log.gz 2024-04-24T18:27:25.778 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58701.log 2024-04-24T18:27:25.788 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72537.log 2024-04-24T18:27:25.815 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.72537.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72537.log.gz 2024-04-24T18:27:25.826 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68875.log 2024-04-24T18:27:25.836 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59091.log 2024-04-24T18:27:25.836 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68875.log.gz 2024-04-24T18:27:25.846 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50078.log.gz 2024-04-24T18:27:25.857 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.59091.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63395.log 2024-04-24T18:27:25.857 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.63395.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60294.log 2024-04-24T18:27:25.879 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.60294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56959.log 2024-04-24T18:27:25.902 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71670.log 2024-04-24T18:27:25.912 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.71670.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.71670.log.gz 2024-04-24T18:27:25.912 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66841.log 2024-04-24T18:27:25.912 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66841.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66841.log.gz 2024-04-24T18:27:25.912 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64439.log 2024-04-24T18:27:25.912 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64439.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64439.log.gz 2024-04-24T18:27:25.913 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.62306.log 2024-04-24T18:27:25.922 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.62306.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.62306.log.gz 2024-04-24T18:27:25.922 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49495.log 2024-04-24T18:27:25.922 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.49495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48874.log 2024-04-24T18:27:25.922 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.48874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70122.log 2024-04-24T18:27:25.932 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.70122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50878.log 2024-04-24T18:27:25.943 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70122.log.gz 2024-04-24T18:27:25.954 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53631.log 2024-04-24T18:27:25.965 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53098.log.gz 2024-04-24T18:27:25.965 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51046.log 2024-04-24T18:27:25.976 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48497.log.gz 2024-04-24T18:27:25.987 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.51046.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65772.log 2024-04-24T18:27:26.009 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.65772.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65772.log.gz 2024-04-24T18:27:26.009 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69756.log 2024-04-24T18:27:26.009 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59091.log.gz 2024-04-24T18:27:26.021 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58701.log.gz 2024-04-24T18:27:26.036 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60817.log 2024-04-24T18:27:26.036 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69756.log.gz 2024-04-24T18:27:26.052 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63676.log 2024-04-24T18:27:26.064 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.63676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70868.log 2024-04-24T18:27:26.069 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63676.log.gz 2024-04-24T18:27:26.127 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50786.log 2024-04-24T18:27:26.127 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70868.log.gz 2024-04-24T18:27:26.137 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50878.log.gz 2024-04-24T18:27:26.159 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68354.log 2024-04-24T18:27:26.173 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.68354.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.68354.log.gz 2024-04-24T18:27:26.174 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.59407.log 2024-04-24T18:27:26.197 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.59407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45738.log 2024-04-24T18:27:26.206 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster2-client.admin.59407.log.gz 2024-04-24T18:27:26.210 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51046.log.gz 2024-04-24T18:27:26.218 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48874.log.gz 2024-04-24T18:27:26.221 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53631.log.gz 2024-04-24T18:27:26.223 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.60294.log.gz 2024-04-24T18:27:26.233 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49495.log.gz 2024-04-24T18:27:26.241 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.45738.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.45738.log.gz 2024-04-24T18:27:26.241 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57980.log 2024-04-24T18:27:26.247 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68072.log 2024-04-24T18:27:26.266 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68072.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68072.log.gz 2024-04-24T18:27:26.266 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.45936.log 2024-04-24T18:27:26.279 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.0.45936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62764.log 2024-04-24T18:27:26.300 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.62764.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62764.log.gz 2024-04-24T18:27:26.300 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67424.log 2024-04-24T18:27:26.311 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.67424.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.admin.67424.log.gz 2024-04-24T18:27:26.311 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47736.log 2024-04-24T18:27:26.321 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50786.log.gz 2024-04-24T18:27:26.321 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59239.log 2024-04-24T18:27:26.321 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.59239.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.59239.log.gz 2024-04-24T18:27:26.321 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46841.log 2024-04-24T18:27:26.322 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46841.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69933.log 2024-04-24T18:27:26.322 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69933.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69933.log.gz 2024-04-24T18:27:26.322 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61527.log 2024-04-24T18:27:26.322 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61527.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61527.log.gz 2024-04-24T18:27:26.322 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50212.log 2024-04-24T18:27:26.332 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.50212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51853.log 2024-04-24T18:27:26.344 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46623.log 2024-04-24T18:27:26.390 INFO:teuthology.orchestra.run.smithi042.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-04-24T18:27:26.412 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.46623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43934.log 2024-04-24T18:27:26.424 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.43934.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43934.log.gz 2024-04-24T18:27:26.424 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55713.log 2024-04-24T18:27:26.433 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46841.log.gz 2024-04-24T18:27:26.439 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46623.log.gz 2024-04-24T18:27:26.450 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51853.log.gz 2024-04-24T18:27:26.450 INFO:teuthology.orchestra.run.smithi136.stderr: 18.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.45936.log.gz 2024-04-24T18:27:26.461 INFO:teuthology.orchestra.run.smithi136.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50212.log.gz 2024-04-24T18:27:26.472 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56959.log.gz 2024-04-24T18:27:26.472 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47672.log 2024-04-24T18:27:26.485 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60817.log.gz 2024-04-24T18:27:26.486 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45626.log 2024-04-24T18:27:26.497 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.45626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51790.log 2024-04-24T18:27:26.531 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.51790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64579.log 2024-04-24T18:27:26.542 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.64579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71636.log 2024-04-24T18:27:26.553 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64579.log.gz 2024-04-24T18:27:26.553 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57980.log.gz 2024-04-24T18:27:26.577 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.71636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53371.log 2024-04-24T18:27:26.602 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.53371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53369.log 2024-04-24T18:27:26.646 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.53369.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53369.log.gz 2024-04-24T18:27:26.646 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54343.log 2024-04-24T18:27:26.658 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.54343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.47215.log 2024-04-24T18:27:26.673 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.47215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63384.log 2024-04-24T18:27:26.699 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.63384.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63384.log.gz 2024-04-24T18:27:26.699 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55360.log 2024-04-24T18:27:26.710 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66646.log 2024-04-24T18:27:26.732 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55713.log.gz 2024-04-24T18:27:26.747 INFO:teuthology.orchestra.run.smithi136.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54343.log.gz 2024-04-24T18:27:26.755 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.66646.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66646.log.gz 2024-04-24T18:27:26.755 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.44303.log 2024-04-24T18:27:26.807 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.44303.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.44303.log.gz 2024-04-24T18:27:26.808 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58743.log 2024-04-24T18:27:26.817 INFO:teuthology.orchestra.run.smithi136.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55360.log.gz 2024-04-24T18:27:26.838 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49051.log 2024-04-24T18:27:26.873 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.49051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64324.log 2024-04-24T18:27:26.884 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47672.log.gz 2024-04-24T18:27:26.903 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.64324.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.64324.log.gz 2024-04-24T18:27:26.903 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50849.log 2024-04-24T18:27:26.911 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51790.log.gz 2024-04-24T18:27:26.920 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71577.log 2024-04-24T18:27:26.931 INFO:teuthology.orchestra.run.smithi136.stderr: 5.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45626.log.gz 2024-04-24T18:27:26.948 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.71577.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.71577.log.gz 2024-04-24T18:27:26.949 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56757.log 2024-04-24T18:27:26.981 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.56757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43230.log 2024-04-24T18:27:27.014 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.43230.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43230.log.gz 2024-04-24T18:27:27.014 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57782.log 2024-04-24T18:27:27.025 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster2-client.admin.53371.log.gz 2024-04-24T18:27:27.058 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.57782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65078.log 2024-04-24T18:27:27.063 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58743.log.gz 2024-04-24T18:27:27.074 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.65078.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65078.log.gz 2024-04-24T18:27:27.074 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70675.log 2024-04-24T18:27:27.096 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.70675.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70675.log.gz 2024-04-24T18:27:27.096 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45850.log 2024-04-24T18:27:27.118 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56757.log.gz 2024-04-24T18:27:27.135 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.45850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60661.log 2024-04-24T18:27:27.147 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.60661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55600.log 2024-04-24T18:27:27.158 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49051.log.gz 2024-04-24T18:27:27.170 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.55600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.57266.log 2024-04-24T18:27:27.193 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.admin.57266.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.57266.log.gz 2024-04-24T18:27:27.193 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53975.log 2024-04-24T18:27:27.205 INFO:teuthology.orchestra.run.smithi136.stderr: 18.1% -- replaced with /var/log/ceph/cluster2-client.admin.47215.log.gz 2024-04-24T18:27:27.221 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50849.log.gz 2024-04-24T18:27:27.232 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.53975.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61932.log 2024-04-24T18:27:27.243 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47736.log.gz 2024-04-24T18:27:27.254 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.61932.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61932.log.gz 2024-04-24T18:27:27.255 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68198.log 2024-04-24T18:27:27.271 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.68198.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68198.log.gz 2024-04-24T18:27:27.271 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68812.log 2024-04-24T18:27:27.284 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55600.log.gz 2024-04-24T18:27:27.292 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68812.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68812.log.gz 2024-04-24T18:27:27.302 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58680.log 2024-04-24T18:27:27.322 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.58680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62088.log 2024-04-24T18:27:27.326 INFO:teuthology.orchestra.run.smithi136.stderr: 4.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45850.log.gz 2024-04-24T18:27:27.337 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.62088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54038.log 2024-04-24T18:27:27.337 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62088.log.gz 2024-04-24T18:27:27.350 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.54038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69954.log 2024-04-24T18:27:27.365 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.69954.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69954.log.gz 2024-04-24T18:27:27.365 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71151.log 2024-04-24T18:27:27.379 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.71151.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71151.log.gz 2024-04-24T18:27:27.379 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50497.log 2024-04-24T18:27:27.390 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.50497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.43843.log 2024-04-24T18:27:27.403 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.admin.43843.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.43843.log.gz 2024-04-24T18:27:27.403 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68854.log 2024-04-24T18:27:27.419 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster1-client.mirror.68854.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68854.log.gz 2024-04-24T18:27:27.419 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47537.log 2024-04-24T18:27:27.491 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cluster2-client.mirror.47537.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60661.log.gz 2024-04-24T18:27:27.820 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53975.log.gz 2024-04-24T18:27:27.820 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54038.log.gz 2024-04-24T18:27:27.821 INFO:teuthology.orchestra.run.smithi136.stderr: 94.3% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-04-24T18:27:27.842 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57782.log.gz 2024-04-24T18:27:27.842 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50497.log.gz 2024-04-24T18:27:27.894 INFO:teuthology.orchestra.run.smithi136.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58680.log.gz 2024-04-24T18:27:28.147 INFO:teuthology.orchestra.run.smithi136.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47537.log.gz 2024-04-24T18:27:33.577 INFO:teuthology.orchestra.run.smithi136.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.admin.71636.log.gz 2024-04-24T18:27:33.911 INFO:teuthology.orchestra.run.smithi136.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.admin.68516.log.gz 2024-04-24T18:27:34.712 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.64383.log.gz 2024-04-24T18:27:34.860 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.admin.65435.log.gz 2024-04-24T18:27:35.099 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.admin.69513.log.gz 2024-04-24T18:27:35.180 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.66414.log.gz 2024-04-24T18:27:35.388 INFO:teuthology.orchestra.run.smithi136.stderr: 93.9% -- replaced with /var/log/ceph/cluster1-client.admin.70542.log.gz 2024-04-24T18:27:35.564 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.62419.log.gz 2024-04-24T18:27:36.333 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.admin.67483.log.gz 2024-04-24T18:27:37.543 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.admin.63395.log.gz 2024-04-24T18:27:58.264 INFO:teuthology.orchestra.run.smithi042.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-04-24T18:28:05.956 INFO:teuthology.orchestra.run.smithi042.stderr: 95.5% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-04-24T18:28:09.421 INFO:teuthology.orchestra.run.smithi042.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-04-24T18:28:09.424 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-04-24T18:28:09.424 INFO:teuthology.orchestra.run.smithi042.stderr:real 0m46.611s 2024-04-24T18:28:09.424 INFO:teuthology.orchestra.run.smithi042.stderr:user 2m5.347s 2024-04-24T18:28:09.424 INFO:teuthology.orchestra.run.smithi042.stderr:sys 0m4.960s 2024-04-24T18:28:19.238 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.46195.log.gz 2024-04-24T18:28:20.709 INFO:teuthology.orchestra.run.smithi136.stderr: 97.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45945.log.gz 2024-04-24T18:28:34.955 INFO:teuthology.orchestra.run.smithi136.stderr: 97.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45974.log.gz 2024-04-24T18:28:43.976 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.48406.log.gz 2024-04-24T18:28:47.332 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.50396.log.gz 2024-04-24T18:28:48.382 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.52360.log.gz 2024-04-24T18:28:48.550 INFO:teuthology.orchestra.run.smithi136.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.admin.47224.log.gz 2024-04-24T18:28:48.659 INFO:teuthology.orchestra.run.smithi136.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.admin.49388.log.gz 2024-04-24T18:28:49.566 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.54376.log.gz 2024-04-24T18:28:51.461 INFO:teuthology.orchestra.run.smithi136.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.admin.53380.log.gz 2024-04-24T18:28:52.822 INFO:teuthology.orchestra.run.smithi136.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.admin.51366.log.gz 2024-04-24T18:28:58.906 INFO:teuthology.orchestra.run.smithi136.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-04-24T18:29:03.950 INFO:teuthology.orchestra.run.smithi136.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.45974.log.gz 2024-04-24T18:29:10.493 INFO:teuthology.orchestra.run.smithi136.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.admin.55393.log.gz 2024-04-24T18:29:12.547 INFO:teuthology.orchestra.run.smithi136.stderr: 96.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.45945.log.gz 2024-04-24T18:29:45.417 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.admin.61423.log.gz 2024-04-24T18:29:56.209 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.60407.log.gz 2024-04-24T18:30:02.461 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.56387.log.gz 2024-04-24T18:30:05.345 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.admin.59416.log.gz 2024-04-24T18:30:10.228 INFO:teuthology.orchestra.run.smithi136.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.admin.57484.log.gz 2024-04-24T18:30:18.780 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/cluster1-client.admin.58445.log.gz 2024-04-24T18:30:41.406 INFO:teuthology.orchestra.run.smithi136.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-04-24T18:30:52.550 INFO:teuthology.orchestra.run.smithi136.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-04-24T18:30:54.923 INFO:teuthology.orchestra.run.smithi136.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-04-24T18:30:54.926 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-24T18:30:54.926 INFO:teuthology.orchestra.run.smithi136.stderr:real 3m32.076s 2024-04-24T18:30:54.926 INFO:teuthology.orchestra.run.smithi136.stderr:user 5m9.724s 2024-04-24T18:30:54.926 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m15.605s 2024-04-24T18:30:54.927 INFO:tasks.ceph:Archiving logs... 2024-04-24T18:30:54.928 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592/remote/smithi042/log 2024-04-24T18:30:54.929 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T18:31:09.797 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592/remote/smithi136/log 2024-04-24T18:31:09.798 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T18:31:48.807 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T18:31:48.818 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T18:31:48.818 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T18:31:48.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T18:31:48.911 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T18:31:48.919 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T18:31:48.920 DEBUG:teuthology.orchestra.run.smithi042:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T18:31:48.922 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T18:31:48.938 INFO:teuthology.orchestra.run.smithi136.stderr:bash: ntpq: command not found 2024-04-24T18:31:48.940 INFO:teuthology.orchestra.run.smithi042.stderr:bash: ntpq: command not found 2024-04-24T18:31:48.946 INFO:teuthology.orchestra.run.smithi042.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T18:31:48.946 INFO:teuthology.orchestra.run.smithi042.stdout:=============================================================================== 2024-04-24T18:31:48.946 INFO:teuthology.orchestra.run.smithi042.stdout:^- hv01.front.sepia.ceph.com 2 8 377 163 +721us[ +799us] +/- 570ms 2024-04-24T18:31:48.946 INFO:teuthology.orchestra.run.smithi042.stdout:^* hv02.front.sepia.ceph.com 3 7 377 28 -334us[ -301us] +/- 57ms 2024-04-24T18:31:48.946 INFO:teuthology.orchestra.run.smithi042.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +454us[ +487us] +/- 83ms 2024-04-24T18:31:48.946 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T18:31:48.977 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T18:31:48.977 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-04-24T18:31:48.977 INFO:teuthology.orchestra.run.smithi136.stdout:^- hv01.front.sepia.ceph.com 2 9 377 218 +818us[ +772us] +/- 569ms 2024-04-24T18:31:48.977 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -347us[ -400us] +/- 57ms 2024-04-24T18:31:48.977 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +470us[ +419us] +/- 83ms 2024-04-24T18:31:48.977 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T18:31:48.978 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T18:31:48.999 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T18:31:49.000 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T18:31:49.009 DEBUG:teuthology.orchestra.run.smithi042:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T18:31:49.085 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T18:31:49.162 DEBUG:teuthology.orchestra.run.smithi042:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T18:31:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T18:31:49.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-24T18:31:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T18:31:49.229 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T18:31:49.239 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T18:31:49.247 INFO:teuthology.task.internal:Duration was 9979.382754 seconds 2024-04-24T18:31:49.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T18:31:49.257 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T18:31:49.257 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T18:31:49.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T18:31:49.317 INFO:teuthology.orchestra.run.smithi042.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T18:31:49.359 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T18:31:49.628 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T18:31:49.628 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com 2024-04-24T18:31:49.628 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T18:31:49.657 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-04-24T18:31:49.657 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T18:31:49.679 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T18:31:49.679 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T18:31:49.700 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T18:31:49.774 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T18:31:49.774 DEBUG:teuthology.orchestra.run.smithi042:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T18:31:49.777 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T18:31:49.865 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T18:31:49.875 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T18:31:49.876 DEBUG:teuthology.orchestra.run.smithi042:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T18:31:49.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T18:31:49.948 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T18:31:49.959 DEBUG:teuthology.orchestra.run.smithi042:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T18:31:49.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T18:31:49.993 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = core 2024-04-24T18:31:50.029 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-04-24T18:31:50.043 DEBUG:teuthology.orchestra.run.smithi042:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T18:31:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T18:31:50.076 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T18:31:50.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T18:31:50.100 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T18:31:50.111 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T18:31:50.112 DEBUG:teuthology.misc:Transferring archived files from smithi042:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592/remote/smithi042 2024-04-24T18:31:50.112 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T18:31:50.169 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-24_14:09:24-rbd-wip-yuriw4-testing-20240423.151325-quincy-distro-default-smithi/7671592/remote/smithi136 2024-04-24T18:31:50.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T18:31:50.209 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T18:31:50.210 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T18:31:50.212 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T18:31:50.259 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T18:31:50.271 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T18:31:50.271 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T18:31:50.281 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T18:31:50.281 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T18:31:50.284 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T18:31:50.303 INFO:teuthology.orchestra.run.smithi042.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 18:31 /home/ubuntu/cephtest 2024-04-24T18:31:50.316 INFO:teuthology.orchestra.run.smithi136.stdout: 262175 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 18:31 /home/ubuntu/cephtest 2024-04-24T18:31:50.318 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T18:31:50.328 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T18:31:50.350 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} workloads/compare-mirror-image-alternate-primary-nbd} duration: 9979.382753610611 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-24T18:31:50.351 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T18:31:50.441 INFO:teuthology.run:pass